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Abstract 


Emerging Methodologies 
for Interdisciplinary Research Practice 

Angus Graeme Forbes 


Media arts practice often ineludes activities that may be at odds with the goals 
of empirically-oriented research, such as experimenting with representation, chal- 
lenging cultural assumptions, and questioning the meaning and impaets of tech- 
nology and innovation. This dissertation investigates the following question: what 
methodologies effectively enable media artists to opérate both pragmatically and 
creatively within interdisciplinary research projeets? To answer this question, I 
propose framing media arts activities in terrns of the following methodological 
thernes: generation, augmentation, provocation , and mediation. I present a series 
of collaborative art works and research projeets— including Data Flow, Fluid Au¬ 
tómata , Information Poems , Annular Genealogy, Natural Material Browser, and 
Augmented Communication Tools, among others— as case studies that elucidate 
these overlapping thernes. I contend that they more accurately describe the Cre¬ 
ative activities of media artists, and moreover that they provide an effective way 
for explaining the emerging roles of the media artist when integrating Creative 
arts practices into interdisciplinary research projeets. 
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Chapter 1 

Media Arts Roles and 
Methodologies 


If the fruits of hybrid research are not strictly Science, or engineering, 
or art, then one must wonder about the epistemological and ontologi- 
cal status of these hybrid forms: What exactly are they? What new 
knowledge do they produce or enable? What is their function in the 
world? - Edward Shanken 


1.1 Introduction 


Media arts practitioners are positioned at the crossroads of technology and culture, 
with hybrid expertise in art production and the creation of new technologies. Many 
academic, scientific, and governmental organizations recognize the importance of 
interdisciplinary collaborations that inelude experts from a variety of disciplines, 
including engineers, designers, scientists, as well as media artists. These collabo¬ 
rations, often referred to as “art-science” projeets, are important because of their 
potential to provide new perspectives and new possibilities to the collaborators. 
For media artists, these inelude: opportunities to work with emerging media that 
might not otherwise be directly available; access to resources and funding not 
normally as readily available to art practitioners; and a greater influence in af- 
fecting projeets that may have far-reaching effeets on research practice and policy. 

Art-science collaborations often make use of some of the skills of the media arts 
practitioner, such as knowledge of multimedia technologies and design methodolo¬ 
gies. But at the same time either de-value or de-emphasize other aspeets, such 
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as the use of aesthetic production as a vehicle to tease out cultural issues and 
mediate between disciplines. Thinking of the media artist largely, for instance, as 
a software engineer, or expecting the media artist to be primarily involved witli 
secondary aspects of the project, such as public outreach, limits the possibilities 
for the media artist to llave a more integrated role. Enabling each team mernber 
to achieve to their fullest potential will result in stronger work frorn all disciplinary 
perspectives. On the other hand, media arts practitioners who see their role pri¬ 
marily as challengers, provokers, and questioners risk positioning themselves as 
outsiders not directly influencing research agendas and activities. 

In this dissertation I introduce a methodological framework that encourages a 
more integrated role of the media arts practitioner; a role that ineludes valuadle 
Creative components of artistic activities. Many of the actual, day-to-day contri- 
butions of media arts are not accurately characterized by the current literature, 
and indeed are not generally well articulated by media artists themselves. This 
hinders the ability of the media artist to make useful contributions in collabora- 
tive projeets, as it is often difñcult for the media artist to explain their role in 
these projeets, either to themselves, to collaborators, or to external agencies such 
as grant foundations or galleries. In particular, the valué of the art-producing 
aspect of media arts is often misunderstood, simplified, and marginalized, even, 
ironically, while it is simultaneously celebrated. 

This dissertation proposes the framing of media arts activities in terms of the 
following “methodological themes”: generation, augmentation, provocation , and 
mediation. I contend that these overlapping themes more accurately describe the 
Creative activities of media artists, and that they provide a more effective way for 
explaining the emerging roles of the media artist. I present a series of my own 
collaborative projeets as case studies that elucidate aspects of these themes and 
wliich inspire a general framework for integrating Creative arts practice into inter- 
disciplinary research and engineering projeets. Along the way to presenting these 
projeets and these themes, I explore the motivations, roles, and methodologies 
discussed by a variety of media artists, multimedia researchers, and media arts 
theorists. Many of these perspectives are contradictory to each other, and, even 
within the scope of a single article, demónstrate overt or unarticulated tensions. 
Sorne unifying threads that tie much of the literature together are an aim toward 
a “meta” awareness, an insistence that process is as important as results, and 
a desire to make progress, not only in research, but also in the way research is 
carried out. Phrases like “meta-design,” “meta-critical,” and “meta-knowledge” 
occur in much of the writing by and about media artists. While these phrases 
have precise definitions in the contexts in which they are introduced, they more 
generally refer to the need for new thinking about how research is conducted and 
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about the importance of exploring the meaning and effectiveness of design_an£ 
evaluaHoTT 

Roger Malina, in his 2010 essay, “What are the Different Types of Art-Science 
Collaboration?,” considers a taxonomy of art-science objectives that inelude: aid- 
ing in community outreach or pedagogy; creating liybrid artists/scientists witli 
dual careers that inform each other; utilizing scientific research to understand 
Creative activities; and producing outputs that are scientific, artistic, or both si- 
multaneously [141]. Many media artists, as discussed below, find that working 
within art-science contexts can be, for various reasons, problematic. But at the 
same time these situations present interesting opportunities to work with tech- 
nologies that would otherwise remain inaccessible. While new ways of practicing 
in these contexts are already occurring, relatively little has been written regardin; 


i 

Is 


these practices, and certainly there is no consensus about appropriate methods 
and tasks in the different approaches. Tliis rneans that media arts practitioners 
often re-invent methodologies or take on roles that are not ideal. Articulating ap¬ 
propriate methodologies for effective art-science collaboration is difficult, as the 
methodologies are not simple heuristics, but rather require flexibility, since they 
are highly project-specific and involve múltiple domains of inquiry. And thus the 
contribution of media arts in art-science projeets can be hard to evalúate. A cen¬ 
tral goal of this dissertation is to shed light on effective methodological approaches 
to media arts practice in art-science and interdisciplinary collaborative contexts. 


O 


1.2 Problem Statement 

In comparison to empirical methodologies used to conduct scientific research, the 
activities of media arts practitioners are less clearly, and sometimes contradicto- 
rily, defined. Media artists function in a range of roles. In addition to produc¬ 
ing art, building multimedia tools, critically evaluating culture and technology, 
and speculating about the future, media artists- perhaps because of this array 
of activities- are increasingly becoming involved in or are leading collaborative 
projeets involving interdisciplinary teams. Media arts often inhabit these roles 
simultaneou¿ly r -aeá - cCre _ torced to reconcilc the tensionsArrthes&qierspectives, or 
tojirjrtoignore them altogether. On the one hand, the history that media^arts 
iíhares with conceptual arts implies a critical role, one where the media artist 
thinks about the ramifications of technology, using art as a space for exploring 
cultural, political, or ethical issues. On the other hand, the engineering aspeets of 
íedia arts projeets steers artists to focus on making or applying technology, se^l 
nigAtovehconfigurations in order to find new Creative possibilities. OfAgjjJrs67"thi s 
duality is nntAri^nTr^Ma^iitati]p anH many m edia artistsM^>--mTTfmó : p to negotiate 
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these opposing perspectives. However, this ontological tensión is more apparent 
in collaborative projects. Media arts projects are often complex, requiring co- 
ordination between people with different perspectives. Moreover, media artists 
are increasingly becoming involved in interdisciplinary “art-science” projects that 
inelude a range of expertise in order to discover innovative Solutions to complex 
problems beyond the scope of a single discipline. In order to participate more 
effectively in art-science collaborations, media artists require a clear set of meth- 
ods, tasks, and terminology that will guide their activities within these types of 
projects. 


1.3 Contributions 



By thinking of the “space” of media arts through using the methodological themes 
that I introduce, it is easier for media artist practitioners to focus on foundational 
aspeets of media arts. These aspeets have a different focus and use different meth- 
ods than either fine arts or of engineering, the two fields that media arts is usually 
seen as being closest to. Although media arts is practiced using a mixture of 
artistic and engineering methodologies, I show that it is more fruitful to tliink of 
media arts methodologies as being different in nature than either arts or engineer¬ 
ing methodologies. In the following sections I describe the space of media arts 
practice (as it is useful to interdisciplinary art-science research projects) botli in 
terms of the potential roles of media artists and of their goals. I use a series of 
projects that I am familiar with, either as the main author or as a collaborator, 
in order to examine in detail the various aspeets of these methodological themes, 
and as a first step to evaluating the effectiveness in thinking about these themes. 
It is my hope that other will also find división of the media arts space into activ¬ 
ities centered around these themes as a useful way to become more effective in 
interdisciplinary research projects. 

In this chapter, I also show that the current ways in which many media 
artists define the space of media artists may not be ideal for collaborative re¬ 
search projects. This is not to say that there is anything inherently problcmatic 
with existing categorizations and methodological approaches, or to say that media 
arts does not have important foci outside of interdisciplinary collaborations. How¬ 
ever, as I show in this chapter, without any methodological foundations, media 
jiEtists are often relegated to less integral positions in art-science projects, such as 
miblic outreacB} or arts focused projects that are more-or-less ornamental. With 
these methodologies, media arts practitioners are able to clearly define their role 
and intentions within these projects. 
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The contributions of this dissertation are as follows: 1) To introduce new 
methodological approaches to help trame media arts practice; 2) To define and 
explain roles of the media artist in collaborative “art-science”/interdisciplinary 
projects; 3) To present case studies as examples for thinking about these inte- 
grated methodological approaches and roles; 4) To provide a comprehensive liter- 
ature review of media arts roles, methodological approaches and specific research 
methods; and 5) To suggest and articúlate new lines of media arts research based 
upon these methodological approaches. Chapter 1 contextualizes the methodolog¬ 
ical thernes and artworks in terrns of both the current and historical literature on 
media arts production and art-science collaboration. Chapter 2 presents sorne of 
rny interactive data visualization projects, including the “Ccll Tango” and “Data 
Flow” projects and examines their possible research contributions, examining in 
particular issues in conducting data visualization research and creating data art 
visualizations. Chapter 3 describes two projects, “Fluid Autómata” and “Annular 
Genealogy,” and discusses how artworks can function to generate novel research 
topics. Chapter 4 describes an ongoing project, “Augmented Communication 
Tools,” as well as the “Natural Materials Browser” [70], a project that evolved 
out of an artistic contribution to the “New Dimites” art project to effectively aug- 
rnent domain-specific volumetric data [20]. Both of these projects are examples 
of how these methodological thernes are effective within interdisciplinary research 
projects. I conduele by discussing how these methodologies can help to explain 
and to promote emerging roles of media arts practitioners in interdisciplinary 
collaborations, and by providing an overview of the IEEE VIS Arts Program. 
This workshop, which I co-chaired in October 2013, brings together visualization 
researchers, artists, and designers to explore potential benefits of collaborative 
methodological approaches to research. 

1.4 Background 

Over the last few years, my work as a media arts practitioner has included creating 
multimedia art installations, designing novel visualization algorithms, creating 
software frameworks to make it easier to visualize dynamic data, and researching 
techniques for interacting with scientific data. Although I work both as an artist 
and as a researcher, and I believe that both of these roles inform each other, I 
have found myself increasingly engaged with research questions but that in sorne 
ways my art production has liad a less primary focus. Another way of looking at 
this might be to say that the way I think about media arts has evolved. Rather 
than seeing artworks as personal investigations and expressions, I more often 
prefer to see a more expansive role for the media arts, one that incorporates 
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múltiple perspectives and many sources of data, that invites participation and 
collaboration, and that contributes to interdisciplinary projects. However, there 
is a definite difference between creating media arts projects that have a specific 
artistic agenda and those that might lead to publications in Computer Science 
conference proceedings. As many authors have noted, and as I discuss later in this 
chapter, the methodologies of art and Science are fundamentally different in nature. 
Art projects tend to be more exploratory, coming to a final incarnation through 
many twists and turns; creating an object or an installation that fulfills a range 
of constraints simultaneously, not all of them necessarily clearly or specifically 
articulated, and thus not easily repeatable. Empirically-oriented projects require 
clearer goals, appropriate hypotheses, and well-articulated evaluation methods. 1 
believe that the flexibility inherent in the process of creating art projects can be 
combined with the rigors of empirical projects. This is not to say that research is 
not Creative, or that art is not rigorous, but rather that a clearer methodology is 
needed to clarify how this flexibility and rigor can be combined in the media arts. 

Part of the reason 1 have been more interested in collaborative projects is due 
to the type of research that occurs at UC Santa Barbara within the Media Arts 
and Technology program. Projects led by the MAT faculty have a large techni- 
cal component, and developing the skills to manage Systems involving advanced, 
complex technologies is explicitly encouraged. It is exciting to be on the cutting 
edge of thinking about novel applications of new technology; on the other hand, 
the time and effort that is required to learn these skills attenuates the ability to 
think more expansively about the mecming of these applications of technology. 
This is especially ironic because often the goal of these collaborative projects is 
in large part to facilítate new ways of thinking about and representing technology 
and data collected via these technologies. Art historians Kristine Stiles and Ed- 
ward Shanken, in their insightful essay discussing the use of technology in media 
arts, “Missing in Action: Agency and Meaning in Interactive Art,” warn about 
this problem, inherent in art that engages with implementing new technology. If 
the only meaning of the art is the fact that it, however adroitly, rnakes use of 
new hardware or software, then it suffers as art; it becomes outdated quickly, is 
seen as trivial, or worse, as being in sorne sense compromised. Stiles and Shanken 
describe how media arts projects might unwittingly serve the “interests of indus- 
try” through popularizing commercial producís and technologies and endorsing a 
“co-opted” ideology of interactivity and agency [204], Elsewhere, Shanken notes 
that even at the “dawn” of art and technology there is a tensión between art 
and technology that continúes today [191]. He explores the early conflict between 
media arts and conceptual art- still ongoing- where the output of media arts 
projects is not considered to be actually artistic or truly meaningful, but rather 
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is tainted with, as explained by Jack Burnham, the “chic superficiality” of “the 
uptown discotheque” [21]. More recent gripes about media arts attack its trivial- 
ity when used to create generative art [7], mathematical art [104], animated films 
[18], video games [53], festival installations and VJ performances [37], and, most 
relevant to this dissertation, art-science collaborations [104, 140, 234]. 

Along witli many others who write about the politics of new media, such as 
Rita Raley and Pamela Jennings, Edward Shanken claims that a primary agenda 
of the mediaaríist-i&-kr be conceptualiy focused and poli licall y -?tw««¿__^Vnd , whilc 
this-beííéTis not necessarily shared by all media artists, it indicates some tensiSr 
that are commonly discussed within the media arts communitv. What is the role 
of the media artist? What agendas are they fulfilling? Is it the media artists’ 
responsibility to raise ethical concerns? Is a purely aesthetic impulse a form of 
fecadence? How do media artists negotiate this tensión between using technology 
anchqtrestiatiingtechnology? One common perception of the media artist]s_r©Ie4s 
that he or she provoke s 7 piublemntiEOG, challeng er — embuda iii^TEéquintessential 
role of the outsider. On the other hand, media artists are increasingly becoming 
insiders, and are considered instrumental in successful interdisciplinary projects. 
But why exactly? As I discuss later in this chapter, the reasons range frorri thcir 
general creativity to their specific ability to think through issues of representation, 
translation, and interpretation. Most of these tensions arises out of a concern that 
the media artist, as someone who uses technology, may also be used by technology. 
For a number of reasons, I think that this is an oversimplified dichotomy, and that 
it is indeed possible and worthwhile for media artists to becorne insiders (that is, 
involved in empirically-focused projects) whilc still maintaining artistic agendas, 
without being solely responsible for community outreach, ethical determinations, 
and public mediation. 

Although Shanken articulates a concern about the potential misuses of tech¬ 
nology for artistic and entertainment purposes, other media artists describe them- 
selves explicitly as makers of new technology, or active researchers into scientific 
projects. Many media artists are deeply involved in the process of building tools 
to support thcir own art projects, or to aid other artists in their work. These 
tools generally involve the writing their own software librarles or frameworks, or 
contributing patches or example code to existing ones. As tool-builders, media 
artists have more control over their artistic output, but may then spend large 
swathes of time in making these tools. Artists have always been involved in mak- 
ing or modifying their tools; but while, to take just one example, the 15th century 
painter Jan van Eyck may have pioneered methods for developing oil paints, these 
enabling technological innovations are secondary to his artistic output [96]. In 
my own work, I often find myself drawn into developing software frameworks for 
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particular projects. For instance, after working on a variety of projects involving 
the dynamic streams of heterogenous data for media arts installations (discussed 
in chapter 2), 1 created a framework called Behaviorism that codified sorne of 
the patterns for managing data and animation that 1 found myself reusing. A 
newer project, Aluminum , is a lightweight C++ framework that supports current 
graphics librarles and was created originally to enable projects to be used in the 
AlloSphere Research Facility [2], but also to allow them to be easily ported to 
mobilc devices. 1 have also created software tools for video manipulation and 
stereographic rendering (the latter in collaboration with Charlie Roberts) solcly 
for the purpose of learning a new technology. That is, software tools are sometimes 
created out of a need, and sometimes created before there is even a need. Tool- 
building requires its own form of creativity, but, as interdisciplinary researcher 
Kaley Middlebrooks explains in her article, “New Media Art: A New Frontier 
or a Continued Tradition?,” it is of a different nature [150]. Creativity in tool- 
building is at once more focused and more expansive. More focused, because the 
main innovations that are solved are micro-lcvel- finding robust and elegant ways 
to solve coding problems- and more expansive because the goal is to enable other 
people to be be Creative through using your tools. Although not always com- 
pletely straightforward, there are much clearer evaluation methods for software 
development that there are for artistic production. Every compilation evaluates 
the success of your code; and it is easy to distribute and gather feedback for soft¬ 
ware tools. On the other hand, tool-building is a more abstract form of media 
artist practice. While it promotes and enables Creative projects that may perform 
explicit cultural and aesthetic engagement, this function is not intrinsic to the 
task of tool-building. 

Many media artists have used scientific investigation and scientific materials as 
a basis for their artwork. Historically, a number of research labs fostered creativity 
and innovation by inviting artists to work alongside scientists [97]. For instance, 
John Whitney, a pioneer in Computer animation, was a resident at IBM [90]. The 
experimental filmmaker Stan Vanderbeek worked for a time at Bell Labs [184], 
And artists, musicians, and composers, including Laurie Anderson, Terry Riley 
and the Kronos Quartet have liad work commissioned by NASA [134], SymbioticA, 
at the University of Western Australia, is an artistic laboratory that over the 
last decade has been dedicated “to the research, learning, critique and hands-on 
engagement with the lifc Sciences” [30]. The artist, maker, and builder Garnet 
Hertz has investigated the use of circuits and Circuit bending as an art form [100]. 
Collaborations by Victoria Vesna and James Gimzewski utilize nanotechnology to 
“blur fact and fiction” in order to create hybrid artistic-scientific installations [88]. 
And artists such as Chris Woebken and Natalio Jercmijenko, who was also an 
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artist-in-residence at Xerox PARC, have created installations based on ecological 
Sciences [50]. 

Media artists are increasingly interested in conducting or enabling scientific 
research. Artists and researchers, such as Stephen Wilson [230], Roger Malina 
[138], Christa Sommerer and Laurent Mignonneau [198], Jill Scott [187], John 
Maeda [136, 137], and many others, have begun to position themselves as inter- 
disciplinary investigators that work on and promote the importance of art-science 
projects. Prevalent examples of what also rnight reasonably be considered art- 
science at UC Santa Barbara inelude: the AlloSphere Research Facility led by 
JoAnn Kuchera-Morin, investigating interaction, sonification, and visualization 
in a virtual rcality environment; George Legrady’s work in computational photog- 
raphy and culturally-contextualized information visualization; Marcos Novak’s ex- 
plorations of morphogenesis; Curtis Roads’ compositionally-focused analysis and 
application of matching pursuit algorithms; Marko Peljhan’s MakroLab; and Ted 
Kim’s development of physical simulation algorithms for use in entertainment. 
While these projects maintain an artistic focus, and are sometime criticized for it 
(see, for example, [164]), they often strive to move into a more integrated role, and 
argüe for the importance of new kinds of Creative thinking in scientific research. 
Projects of this scope require múltiple people with different skills, or even large 
teams to function successfully. Regardless of the size of the projects, art-science 
research is inherently collaborative. 

My own collaborative work in research-oriented projects has fallen into differ¬ 
ent areas, including user interface design, knowledge representation, visualization, 
interaction, and software development or database design. For instance, I worked 
with the computational linguist, Stefan Gries, to build the Corpus Browser, a tool 
that allows linguistic researchers to quickly test hypothesis about language using 
a large corpus of written text [71]. I have also worked on several projects related 
to social network visualization and analysis with Saiph Savage [77, 185]. I am 
also continuing to collaborate with Tony Fast, a materials scientist at the Georgia 
Institute of Technology, on a system to share and annotate volumetric materi¬ 
als Science samples [70] (described in Chapter 4). More recently, I have begun 
working with members of Elena Plante’s Speech and Hearing lab at University 
of Arizona on a project to visualize and interact with temporal clusters of brain 
activity that change over time during language acquisition tasks [79]. I am also a 
co-PI on a grant to study cyberinfrastructure for a comparative genomics tool, led 
by Eric Lyons at University of Arizona. Most recently, I am a co-PI on a pending 
grant with the healtli informatics researcher, Jane Carrington, to explore the use 
of new visualization and interaction techniques to improve nurse communication 
via augmenting electronic hcalth records [78] (also introduced in Chapter 4). In 
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fact, the majority of work I do, whether they are part of art or research projects, 
are intrinsically collaborative, as is true for rnost media artists. Media artists are 
thus in a good position to leverage their skills and interests to make important 
contributions to these projects and to wrestle witli many of the issues that arise 
at this intersection between art and research. Throughout all these collaborations 
with domain-experts, and whether or not my role was more or less technical or 
Creative, various dichotomies aróse again and again. Some of these tensión that I 
liad to wrestle with inclnded: an expectation to focus on pnblic outreach while at 
the same time contributing academically; having an empirical ontlook but also an 
interpretive one; having technical, engineering skills at the same time as artistic 
goals; being an insider to the main concerns of the project but also being aware 
of cultural outputs. 

While these tensions may be overly simplistic, they nonetheless indicate a 
prcvailing mindset about the proper role of media artists for particular activities. 
ft is these experiences of needing to articúlate my “hybridity” that has led me to 
this study of media arts methodologies. In the following sections, I will survey 
how other media artists in art-science collaborations have resolved or at least 
come to terms with these tensions, exploring how they work and how they present 
themselves to the world. 

1.4.1 Terminology 

Media arts is at the center of various discourses; this section clarifies some common 
terminology in order to focus the following discussion about the roles and methods 
of media artists. 

Disciplinarity In an often-cited article, the public-health scholar Patricia Rosen- 
field describes “a simple taxonomy” of cross-disciplinary research: multidisci- 
plinary work involves researchers working independently, but to address a common 
issue shared by múltiple disciplines; ínter disciplinar y work involves researchers 
from different fields working together, but from different perspectives; and trans- 
disciplinary work involves a “shared conceptual framework” where researchers col- 
laborate together to merge disciplinary-specific theories, concepts, and approaches 
[178]. For the purposes of this thesis, the terms interdisciplinary and transdisci- 
plinary will be used more-or-less interchangeably, with complete transdisciplinar- 
ity seen primarily as a potentiality that rarely exists in working groups due the 
time and effort it takes to merge disparate perspectives and methodologies. Rosen- 
field writes that transdisciplinary research “has the potential to move [...] to a 
stage where disciplines can build on their distinct traditions and coalesce to be- 
come a new field of research, such as has happened in molecular biology.” Media 
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arts is an example of a field that emerged in part out of distinct traditions of art 
(e.g.. conceptual art, art-and-technology, composition, film) and technology (e.g., 
Computer Science, electrical engineering). 

Media Arts Practitioners who identify themselves as being involved in media 
arts, or some closely-related titlc, such as “intermedia” “new media arts,” “inter- 
active art,” “digital art,” may perform quite different functions. This dissertation 
explores the various roles the media artist inhabits, the communities he or she 
belongs to, and, in part, seeks to investígate what a useful definition of the media 
artist might be for successfully participating in interdisciplinary projects. Part 
of what I will be arguing is that the role of the media artist is precisely that of 
mediating between disciplines to help makes projects more transdisciplinary in 
nature. Since the phrase media artist refers to both artists who use technology as 
wcll as to researchers who explore multimedia technologies (or to both simultane- 
ously), I use the phrase “media arts practitioner” to broaclly indicate this range 
of roles, unless otherwise specified. One definition that I do not discuss is media 
arts as purely visual or aural production and/or effects. While these activities are 
interesting and complex, in general they are used to serve entirely different forrns, 
such as video games or animated films. 

Art-Science The interdisciplinary projects I discuss all involve in some way the 
participation of a media arts practitioners in interdisciplinary research collabora- 
tions that inelude a significant scientific and/or engineering focus. There are of 
course many examples of media artists working with engineers to craft components 
of artworks. Similarly, media artists often appropriate scientific material for artis- 
tic projects in various ways. However, here I specifically refer to projects in which 
the output of the collaboration is neither purely artistic ñor traditionally scientific, 
but rather have some kind of hybrid result that has a broader result of interest to 
both artists and scientists. These types collaborations are called by various ñames, 
such as “art-science” project, “transdisciplinary” research, “STEM” or “STEAM” 
initiatives, all of which indicate a particular type of research that require the exper- 
tise of múltiple disciplines and aim to broaden the scope of how certain problems 
are solved or research tasks are thought of. I use all of these terminologies- usu- 
ally “interdisciplinary” or “art-science” collaboration- and point out particular 
nuances as needed. 
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1.5 Motivation 

In this section, I draw a trajectory from general to specific reasons for the impor- 
tance of media arts practice, especially as it relates to interdisciplinary collabo- 
ration. Works of media art tend to be evaluated either in terrns of cultural or 
pragmatic utility. Media arts is often regarded in generalist terrns that describe 
its societal contribution in creating producís of cultural enrichment, tools for pro- 
moting innovation, or provicling the rneans by which to think critically about the 
ethical ramifications of technology. Media arts is also characterized as having the 
potential ability to aid in the solving of specific scientific and engineering prob- 
lerns, especially those having to do with Creative representations of, interacting 
with, and reasoning about data. At all levels of this trajectory, there is a struggle 
to articúlate concerns regarding the tensión between artistic outlooks and tech- 
nical engagement, as wcll as to define the role and methods of the media artist 
within both art and research contexts. 

The importance of media arts as an art form is often explained in a similar 
manner as other arts- they imbue society with vibrancy and culture. Jennifer 
Craik, a researcher who explores comparative cultural policy, for instance, ad- 
dresses this in Re-Visioning Arts and Cultural Policy [40], exploring the reference 
to “cultural vitality” as a prevalent justification for the governmental funding 
of media arts. More specifically, Martha Nussbaum, the American philosopher 
and ethicist, in her book, “Not for Profit: Why Democracy Needs the Humani- 
ties,” sees a correlation between creativity and democratic principies, exploring 
how necessary ingredients for a robust society, such as the ability to empathize 
with others, to see problems from múltiple perspectives, and to think critically 
and ask questions, are effectively learned through a strong arts curriculum [161]. 
The artist and researcher Pamela Jennings also describes the societal benefits of 
media arts: “The new media arts express a risk-taking and subversive attitude, ul- 
timately seeking cultural acts through which to provide society with entry points 
for change” [106]. Seen in these ways, media arts practice as a whole is a humanis- 
tic endeavor that makes Science more accessiblc and increases public engagement 
with issues surrounding technological innovation. 

At the same time, many research initiatives point to changes in the landscape 
of STEM (science, technology, engineering, and mathematics) research that indi- 
cate that new ways of thinking creatively and collaborativcly are becoming increas- 
ingly important for various practical reasons. A study released by The Conference 
Board in 2008, “Ready to Innóvate,” explores the relationship between creativity 
and innovation, and argües for the importance of instilling creativity via educa- 
tion in order to foster a globally-competitive workforce [130]. A recent report 
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released by The President’s Committee on Arts and Humanities specifically out- 
lines recommendations to facilítate the promotion of arts education, including an 
emphasis on integrating art practice with other subjects, which seerns to improve 
overall academic involvement and to enconrage innovative thinking [171]. John 
Maeda also describes the need for integrating the arts into government-backed 
STEM initiatives so that they are recognized as being “part of the same pool 
of knowledge.” More hyperbolically, he explains that because art is needed to 
hclp foster innovation it is therefore is “vital to national security” 1 . Specifically, 
the STEM to STEAM Initiative (championed by Maeda) presents a series of case 
stndies involving experimcnts in early education and corporate collaborations in 
creating and packaging “design-centered and human-focused products” 2 . 

Other authors more concretely examine contemporary needs that reqnire cre- 
ativity and innovation and explore how the media arts (or Creative aspects of 
Computer engineering) specifically can be effective. For instance, Roger Malina 
outlines increasingly problcmatic issues in contemporary scientific practice and 
hopes that art-science collaboration may lead new techniques and methodologies 
for approaching them. In a book review pnblished in Leonardo, he explains that 
a primary motivation for the “art-science movement” is the “epistemological in¬ 
versión” that has occnrred dne the exponential increase in data that is available 
whereby the Sciences have become “data rich and meaning poor” [140]. Moreover, 
the type of data has changed- Instead of having “snapshots” of data, there are 
temporal “streams,” or complex ”data Systems.” This lcads to a “crisis in repre- 
sentation.” Science is largely conducted throngh data analysis and not throngh 
empirically sensing the world, and, because there are no obvious ways to represent 
probabilistic, fleeting, or otherwise nnintnitive data, this necessitates the attention 
of the artistic sensibility that is skilled at thinking about issues of representation, 
or what he calis “re-sensing”: “use of visualization and sonification technologies 
from compnter Science are a rich terrain of art-science practice and shonld be 
viewed as projects in translation” [139]. Moreover, this “delnge” of data creates 
potential “blind spots”- areas and interpretations that are ignored because the 
amount of data exceeds our ability to analyze it. Finally, he advocates for the 
importance of art-science collaboration because the Solutions to complex, global 
issues require diverse skill sets and the involvement of experts from múltiple disci¬ 
plines. Interestingly, Malina disagrees with the idea of a transdisciplinary merging 
of art and Science, but instead imagines a “shared language” that “entails shared 
ontologies and eventually connected epistemologies” that “contribute to creativity 
and innovation.” However, the process of building this shared language involves 

1 http: //www .ne wscientist. com/blogs / culturelab /2012/08/j ohn-maeda-steam. ht mi 

2 http://www. stemtosteam.org 
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a perpetual negotiation between different perspectives, a process of “trade and 
barter and not assimilation” [139]. 

Another motivating factor for this dissertation is the continued participation 
of media artists in developing tools and platforms that foster Creative ways of rep- 
resenting and interacting with data. In a 2007 essay, Ben Shneiderman, a pioneer 
in the field of information visualization, describes the need for tools that enable 
researchers to come to new understanding or to rnake discoveries. These tools 
involve the various stages of research, frorn the “early stages of gathering infor¬ 
mation, hypothesis generations, and initial prodnction throngh the later stages 
of refinement, validation, and dissemination” [195]. Specific examples of tasks 
that creativity support tools facilítate inelude: the rapid generation of múltiple 
alternatives, the freedom to explore implications of alternatives, and the option to 
revert to earlicr stages. Linda Candy, co-founder of the ACM SIGCHI Creativity 
and Cognition conference series, sees the media artist as especially wcll sitnated 
to develop new creativity support tools. She explains that media arts projeets 
present a way to “harness” the various “complex social, organizational, and cul¬ 
tural factors” that are required for innovation. Candy thinks of the artist as a 
“power user” that pushes the boundary existing frameworks and tools in order to 
discover new forms of expression [23]. Zafer Bilda, an Australian researcher in 
design cognition, also examines media arts installations as proving grounds for 
techniques that are helpful for understanding how different forms of interaction 
promote increased engagement and creativity [13]. Pamela Jennings further ex- 
tends the idea of creativity support tool as a framework for pervasive computing. 
She explores ways in which media artists more expansively consider creativity sup¬ 
port tools to be “distributed structures that mutually reinforce both individual 
and social creativity.” She recognizes that any tool is part of “a socio-technical 
architecture deeply interwoven with the physical environment and social fabric of 
local communities” and calis for a more “pervasive” Outlook on creativity when 
“investigating and promoting situated and distributed aspeets of creativity, partic- 
ularly in relation to temporal, spatial and conceptual distribution across múltiple 
interaction spaces” [106]. 

Edward Shanken, in “Artists in Industry and the Academy: Collaborative 
Research, Interdisciplinary Scholarship and the Creation and Interpretation of 
Hybrid Forms,” explores the gap between the perceived importance of media arts 
engagement with Science and technology and lack of clear methodologies for con- 
ducting interdisciplinary projeets [192], He writes that “despite the general recog- 
nition that there are substantial challenges to collaboration across disciplines, 
there is scant metacritical research that studies best practices, working methods, 
and contextual supports and hinderances.” Moreover, there is confusión about 
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the end goals of Ínterdisciplinary projects, especially when it does not lead to a 
typical artistic or scientific outcome: “one rnust wonder about the epistemologi- 
cal and ontological status of these hybrid forms. What exactly are they? What 
new knowledge do they produce or enable? What is their function in the world?” 
Shanken is concerned that if there are no clear practices or results to evalúate, 
then artists and researchers will not have the incentive to “facilítate the creation 
of new forms of invention, knowledge and meaning.” On the other hand, there are 
exemplary projects that rnight serve as case studies, and certain individual artists 
are successful at functioning within interdisciplinary contexts. But it is less clear 
exactly how and why they are are able to function as “catalysts.” That is, it is not 
always clear what specific methods were used in these projects and what specific 
roles media artists take on in order to help make them successful. 

Media arts practice, with its hybrid focus on art (thinking about, extending, 
and challenging representation) and engineering (creating new Systems, making 
use of new technologies) is in an appropriate space to begin to tliink about the cul¬ 
tural and technological concerns and opportunities inherent in shaping technology 
for new research methodologies. In the next section, I explore a sampling of the 
various roles that media arts have defined for themselves, especially in relation to 
interdisciplinary collaboration. 


1.6 Media Arts Roles 


As described above, the discipline of media arts ineludes a wide range of practices, 
with a wide variety of roles available to different media artists. Nonethcless, in the 
literature describing the roles of media arts practitioners a few general categories 
emerge, especially in the context of participating in interdisciplinary projects. 
These roles inelude diverse activities sucli as: managing public outreach, leading 
interdisciplinary teams, organizing politically-charged events, building software 
tools, as well as making art objeets or installations. In this section I look at 
sorne representative examples of these roles drawn frorn ethnographic studies of 
art-science collaborations, essays on pedagogical approaches, and theoretical ide- 
alizations of the media artist. Different methodologies follow frorn these different 
roles, and I argüe that all of these conceptions indicate an evolution of the ac¬ 
tivities of the media artist, and that, taken as a whole, they indicate interesting 
tensions in the perspectives and goals of the media artist. 
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1.6.1 Art-Scieniie-GellabüraüoiT 


Q 


In the essa]^j“Art and Science Research: Active Contexts and Discourses," tne 
Switzerland-hcCsetLartist^ Jill Scott and Daniel Bisig describe theirexp^rielíces 
during recent collaborativ ñ ~ inLcldisuplinary pr u jects and l ' aise~ ~ q uestions regard- 
ing the effective role of the media artist. They conclnde that inedia artists who 
work in collaborative contexts with scientists are provided witli the materials to 
conduct “critical analysis in the pnblic realrn” and believe that through these col- 
laborations media artists can “perhaps even conduct social change.” For Scott and 
Bisig, the primary motivation for bringing together media artists and scientists 
or engineers is so that the activities of researchers can be better explained and 
presented to the public. That is, the skill set of media artists positions them 
to be effective mediators between the academy and the public, and to construct 
effective, educational p ublic outreach campaigns. Internallv to the collaborative 
projectj^ihe-rrincílá^artist, as an artist and simply as an ent husiastTtrrmteid^, is 
so able to: challenge disciplinary hierarchies to promote new insights, to criti 
cally examine the prevailing tacit methodologies, to point out that knowledge is 
sitm&ted in culture and socially constructed, and to raise ethical considera!' 
[188], 

These examinations also discuss how artistic methodologies may promote Cre¬ 
ative thinking and how creating non-hierarchical researchjcenters may help foster 
sharing and communication. However, the authors think that it is the media 
artist’s general outlook that is useful to the collaboration, rather than a partic¬ 
ular set of skills. Because artists have skills at finding unique visual metaphors, 
are not afraid to say what they think, and because are concerned with artistry of 
communication, they envision media artists as the “main players for interpreting 
scientific research” to a general public who find Science “ethically suspect.” They 
introduce the idea of the media artist as an activist who is able to méchate be¬ 
tween the world of research and various public works: “some artists can assist 
local communities in a move toward sustainable developments, others can look for 
ways to raise public awareness” [188]. Though many aspects of their account are 
intriguing, in my experience media artists do not position themselves in terms of 
public outreach and activism when participating in collaborative research environ- 
ments. For instance, one assumption that the articlc makes is that scientists and 
engineers do not consider ethical issues, but they provide no evidence that media 
artists would for some reason have enhanced ethical sensibilices. Although media 
artists often present their work to the public in gallery and museum settings, they 
generally do not act as spokespeople either for the public or for scientists- a role 
that seerns perhaps a more appropriate_iox-ar^ouinalist, whose job it is tu it3p©i±_ 
information to the public. Whilc/Tnere is a rich history of the use of art as a 


O 
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form of activism in media arts (and conceptual art in general), this role tends 
_outsider’s perspective, and this kind of provoking an d/or comm; 
involvement does not necessarily lend ítseil to successtul collaboration. On other 
liand, it does partially capture the spirit of why collaboration is considered im- 
portant. Both scientists and artists are in some sense public figures, working in 
public institutions, tasked with contributing to society. In some projects it may 
rnake sense to utilize the skills of the media artist to mediate between the public 
and the researcher, but in general, it is not characteristic of the ideáis and the 
dav-to -day practi 

ledia artists often choose to be insiders, or even leaders, with a role 
_more fundamentally integrated with the scientists and engineers. The 
C ^Geor ge Leg rady j jpcscnls a series of ai Lides LliaL examine his own work both as 
the leader of a team of interdisciplinary researchers and also as an educator inves- 
tigatijig-Jíiethudtí foi tostermg creativity in interdisciphnary plojecLs.-^JX_article 
co-authored with Brigitte Steinhcider, “Interdisciplinary Collaboration in Digital 
Arts: A Psychological Perspective on the Production Process/Mcontei 
alizes the process of creati ng a cumplex multimedia inslallation both in terms of 
the final artistic output and in terms of the collaborative work needed to create 
the installation [202], The piece, Pockets Full of Memories, applies information 
technologies and self-organizing algorithms in order to conduct a public cultural 
exploration of everyday objects [128]. In particular, Stcinheider and Legrady 
examine some of the practical considerations of working with a diverse team of 
Computer scientists, designers, and artists. Whilc the project resulted in a series of 
successful installations, various problems arouse during both the design and instal¬ 
lation pilases of creating the piece. Tliey explore the tensión between production 
and experimentation; getting tliings done versus remaining open to new possibil- 
ities. From the artist’s perspective, flexibility is necessary in order to maintain 
creativity and to make sure that the final project has aesthetic integrity, despite 
the potential to cause some amount of disappointment (if plans need to change 
or people need to leave or work on different tasks) and inefficiency (as work that 
does not end up fitting the final project must be re-done). However, identifying 
issues with “communication, coordination, and knowlcdge-sharing” may make it 
easier to collaborate. In particular, they emphasize the importance of knowledge- 
sharing. All participants in the project need to have a “meta-knowledge” that 
allows them to connect “the different areas of needed expertise as they relate to 
the project that has brought them together.” 

Another article by George Legrady explores difficulties that arise when media 
artists and engineers attempt to collaborate in a more integrated manner, shar- 
ing in lcadership of a project. In “Perspectives on Collaborative Research and 
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Education in Media Arts,” Legrady describes co-leading an innovative graduate- 
level course with an engineering professor. The course included students from 
different fields who were tasked with working together on conceiving and creating 
novel interdisciplinary projects. In observing the development of these interdisci- 
plinary projects, the course leaders noticed different roles that tend to be played 
by the artist and engineering members of the projects. For instance, artists fo- 
cus on “cultural aspects of media,” look for various kinds of “subtext,” and are 
comfortable with noisy data while, on the other hand, engineers are interested in 
“problem-solving opportunities,” are more comfortable with “purer signáis,” and 
seek reliable, measurable collections of data. Artists are by nature “generalists” 
who incorpórate a “wide spectrum of sources” to come up with a unique approach 
that cannot be reduced or replicated without dissolving “essential qualities.” Ulti- 
mately, Legrady concludes that these differences in approach are somewhat detri- 
mental to engineering goals and that there is not necessarily an immediate benefit 
when including artists in specific, existing research problems. However, he also 
believes tlnt the a i l. i sL.- i insighr, mav "have an ímpact dow n 1.1 u mu il L v op oninp- 

up new vistas” [126]. In his article, “Expectations of Scientists Towards Interdis- 
^iplinary and Transdisciplinary Research,” geographer Marc Antrop also explorg 
some com p T icaLions inherent in finding a joint languago bctw c cn pai ' LicI pañts from 
different domains working on solving interdisciplinary problems. He bclieves that 
interdisciplinarity is actually “an attempt to re-introduce lost skills” that disap- 
peared due to the narrowing of disciplinary focus. In particular, he finds that 
communication with quantitative scientists is difhcult because they do not always 
understand that research can be “holistic, dynamic, and multi-scale” [5]. He en- 
visions a “global theoretical basis” for “a cornmon language understandable and 
meaningful” to a participants in interdisciplinary projects. 

These sarnple case studies present four different perspectives characteristic of 
recent writings describing roles of media artists, as artists, when working in collab- 
orative projects. There are a variety of different configurations of collaborations 
and different tactics for operating in these configurations are still being explored. 
Although a goal may be to have integrated teams of artists and researchers, no 
“best practinesjLhaw-bccn idcnlmed xhat are complelely effectivnT--Haaíeyer, some 
hjiperTtant issues are acknowledged, such as the need for clcarer connnunicaFit 
and knowledge-sharing. The concept of “meta-knowledge” is especially interesting, 
as it encapsulates the importance of liaving each participant be aware of the roles 
leother participants as well as the overall methods and goals of the projecj 
Since theArmrHfíT-ax tjst is p osit ioned in betwee n art and in 

facilitating this meta-knowledge may be important for successful interdisciplinary 
projects involving múltiple participants. 
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1.6.2 Aesthetic Computing 



O 


A different role for the media artist is described irkPaul Fishwick’s “Aesthetic^ 
Computing” [64], He eschews the dichotomies of the artfet- and oclcn t ré TToles 
and does not rv pl n rr t i f i m i I ¡ Ll I Ll ‘U ld Ll¿ ‘i' i nl i 1 - 1 L h ■ ■ i nr it n q lanernao-p 

Ththe contrary. he bclicves that media artists should not aim merely to imer=~- 
ictwith scientists and engineers, but instead think of themselves as scientists 

and7or~engrrrr^i ir i thcmsclvcs_ Tn particular. he i mamnpg mn-lí-i nrtin+r r -ag~?T7gacrprl 

in thinking about the aesthetics of computational representation as it applies to 
data structures, algorithms, as well as user interfaces. His main contention is that 
there is already an embedded sense of aesthetics in mathematics and Computer 
Science that involves invariance, parsimony, minimalism, proportion, and symme- 
try but that framing research in terms of other kinds of aesthetic modalities, such 
as pliability, fluency, and seductiveness could lead to new breakthroughs and also 
help the research to find a wider audience. He finds that the increasing power 
and ubiquity of computers introduces new aesthetic possibilities that can take ad- 
vantage of customization and personalization and engage with issues of modality, 
quality, and culture. Media artists are well situated to think about cultural issues, 
and ]thevj dso can think about new metaphors with which to frame “notational 
mees.” 

Fishwick/Üescribes activities of the computationally and aesthetically engaged 
media artist that ineludes “representing programs and data structures with cus- 
tomized, culturally specific notations,” “incorporating artisti c methods in typically 
comptttirigAntensive activities,” and “improving the emotional and TñThrmd-díiyc 1 
of interaction with the Computer.” That is, the media artist is first and foremost 
iTcompo ifcr scicntist . but with a background in aesthetics a nd cultural m edffrtioíu 
Additionally, Fishwick does see a role íor the media artist in “non-utilitarian” 
roles. A project that is wholly aesthetic, and not directly research-oriented or 
product-oriented, can still influence utility by wrestling with issues of aesthetics, 
modality, metaphor, and culture. Tliese art projeets can llave both a inspirational 
and pedagogical function. 

Other researchers similarly see their aesthetic work as a catalyst for advances 
in the development of Computer programming techniques and frameworks. For 
exan^gle-, Llie ¿lgoriLlimic composers Alex McLean and ueraint Wiggins-disguss 
the ideaol " bncolag e” piugiamming a3 an approach whuiuby Lile conslrálñts and 
opportunities of a particular computational artwork can lead to novel program¬ 
ming met hods. Bricolagc implics a more iterativ e way of thinking, of creating 
^-ddlToñgh crafting, of experimentation based on liniit ed'TeKrmíx:es. In the article, 
^S-h4n ahi,£>e Programm ing in the Creative Arts” thev mak e-arr?tse for thinking more 
metaphorically about programming in order to foster both artistic creativity and 
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computational flexibility. Whilc “shallow” ideas that introduce spurious concepts, 
techniques and systems can be problematic to Computer scientists (cf. Dijkstra’s 
Tntlictmg” misguided “radical novelties [49]), cieatívt 
"working on specific artistic projects are able test out new ways of thinking aboñf 
jtaphors, workflow, and design patterns that are actually effective at promotjj 

pffiripnF~pr7nn~ .. I, , 1-iniqnps [ 147]. Th at, is bv íwinrr nm ii.c."J.lrrfíTHripriR as a 

springboard for programming innovation, Creative practitioners are able to intro¬ 
duce notoal^MiewTtteas and new techniques, but also new ways^fAhinJdng about 
research methods themselves. For example, in “Metaphors We Program 
Alan Blackwcll examines a series of social, physical, and conceptual metaphors [14] 
iderpinning the structure of the Java programming language that may influengf 
the^neWraA-ínxidelsof programmers. One role of the media artist, isJ ^4rrttoríííce 
new metaphors and to expose the metaphors that are aiready being utilized. That 
is, the aesthetic Computer scientist, or the computational media artist can have a 
dual role crcati na- m 1 w idi'.in .ilid proBIémaElzlñg exis 

um Kuchera-Morin also sees a role for aesthetics in Computer Science" 
search. In “Using the Creative Process to Map N Dimensions: Quantum Informa¬ 
tion at Your Fingertips,” she envisions the role of the Computer scientist as similar 
toYhBh-oLamomposer. In particular, she cites the compositional process 
an appropriate m elh o dological systeni for thinking abou t how tomove between 
and intégrate processes occurring at different hierarchical levels. This aesthetic 
process works whether it begins from a “microstructure” or at the “macro-level” 
- with the right tools, a researcher can “unfold” a complex process, creating a 
“navigable immersive context” that can benefit scientific research [119]. The me¬ 
dia artist is thus a composer of data, and uses technology as a “large, multi-user 
instrument” that creates outputs that are simultaneously immersive research en- 
vironments and “abstract art forrns and visual/sonic installations.” 


1.6.3 Post-Media Aesthetics 

In opposition to the more utilitarian agendas prescribed by the above roles in 
which the media artist develops or assists practical research agendas, some writ- 
ers imagine a more inspirational role. For example, in “Liquid Architecture in 
Cyberspace,” Marcos Novak describes the media artist as a visionary architect of 
a digital environment that is defined by a constantly metamorphosizing poetics 
where data can be rnapped to form in an infinite variety of ways [160]. Novak 

advocates a methodology that encourages “minimal restriction” and “maximal 
binding” so that new representational schemas can allow different “correlations” 
to appear within the same data. As Randall Packer and Ken Jordán explain, this 
visionary outlook implies “a commitment to forrns of media and Communications 
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that are non-hierarchical, open, collaborative, and reflective of the free movement 
of the mind at play” [163]. Deleuze and Guattari’s influential concept of the rhi- 
zome as a decentralized organizational strncture is apparent in the mnch of the 
writing about multimedia technology. Tliey conceive of the rhizome as a kind 
of “abstract machine” that “ceaselessly establishes connections between semiotic 
chains, organizations of power, and circumstances relative to the arts, Sciences, 
and^soeiaf-struggltís ' [4GJ 

Lev Manovich, in his article, “Post-Media Aesthetics” also envisions a new role 
for media arts as creators of “information interfaces” that influence societal “in- 
f^f^yftrnTd^h^^inrs ” This concept of media arts subsumes both tradhfi'^TrTTHTííb 
digital media by reframing artwork in terms of its “operations.” For instance, he 
likens the films of Eisenstein to Tim Berners-Lee’s invention of linked multimedia 
pages in that they both challenge the way information is experienced. He theorizes 
that post-media categories can help us think about dimensions of cultural commu- 
nication that might otherwise be hidden. In his conception, the role of the media 
artist then is to invent new experiences- to find new ways of encoding information- 
and in so doing shape new behaviors [143]. Manovich also discusses the fact that 
existing categories of media arts do not represent the actual function of media art, 
and that the emphasis on the new forms of media as material is misplaced. In 
fact, media arts either uses material in “arbitrary combinations” or in fact aims to 
“dematerialize the art object.” Edward Shanken also explores this tensión between 
the media artist’s use of new materials and the their use of concepts that do not 
rely on any materials at all. In Art in the Information Age, {Shanken explores 
how “cybernetics, information theory and Systems theory [are] foundational in- 
tellectual models” that have shaped culture in a significant way, alongside “the 
advent of digital computing and telecommunications.” In particular, he explores 
the cióse ties between media art involving technology and conceptual art [191]. 
In so doing, he teases out interesting issues inherent to the media arts, both in 
relation to aesthetics and technology. He looks at some of the earliest successful 
art exhibitions that involved software and information technologies and finds that 
they involve “meta-critical” practices which privilege a certain type of thinking 
that “challcnges modernist aesthetics and technocracy.” He rnakes a distinction 
between art that uses technology as a spectaclc, emphasizing its materiality, from 
art that proposes that require viewers “to investigate the cognitive functioning of 
their own minds with respect to the processing of information and the creation of 
meaning.” That is, although media arts projects make use of technology, one goal 
of media art is to cause the viewer to think about the nature of certain aspects of 
technology itself and its relationship to the individual and to culture. 
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1.6.4 Tactical Media 


O 


o 


Another^xt4ekrb}TSTianken explores a less idealistic aspect cTmCrfe-miaMn Miss 
Tgui Action: Agency and, Meaning in Interactive Art, he describes how me 
arts (and specifically interactive digital art) can serve the “interests of industry” 
through popnlarizing commercial producís and teclinologies and endorsing a “co- 
opted” ideology of interactivity and agency. By offering “imagined personal con- 
rglto consumers” artists simplify the complex discussions of agency and meani, 
reduchíg'paáiicipaBon to a simple choices constrained by a “hegum omfr^'tr^hetic 
[204]. In some sense. he ís chrecrly cruiqumg rhe mvoivement of artists with re- 
search into the areas Fishwick wants to claim as aesthetic (e.g., HCI, scientific 
visualization, VR). Shanken suggests that instead of “earnestly pursuing techno- 
logical enhancements of agency” artists could take an ethical stance and “focus 
attention on deconstructing the vast ideological apparatus that enlists individuáis 
in their own subjugation.” Art that utilizes technology in a non-critical way, i.e., 
that blindly accepts the benefits of technology, is in some sense being used by 
the technology, or is fulfilling some other, non-art function. While the language 
of subjugation and hegemony indicate the need for successful technological and 
interactive art to involve some level of activism, this can take the form of creating 
interactive works that promote awareness and encourage thinking and reflection: 

“the interactive features of multimedia become meaningful when they engage and 
activate complex emotional and decision-making responses” that reinforce “trans- 
formative effeets” and play ”a constructive role in Creative change and exchange.” 

In “Art, Technology, and Institutional Discourse,” contemporary artist Jill 
Fantauzzacoffin notes that art necessarily has a “destabilizing” role, and that 
“incorporating art and technological innovation practices means admitíing the 
dynamÍ£s-of-^tírBTlIzation and destabilization within the cliscouIlS~TjfdsnQyation” 
Rita Raley’s “Tactical Media” explores how one role of media artists is 
find provocative ways to promote changes in thinking. Effective works of surveil- 
lance art, bio art, performance art, and hacktivism can function to expose cor- 
jorate/capitalistic structures, and to make a space for citizens to think clearR 
politibcdlyy^atidhumanistically. She discusses media arts as disturbagv^A ^ cfical 
media signifies theiu üei veüliun and duruption of a duminanT sémiotic regime, the 
temporary creation of a situation in which signs, messages, and narratives are set 
into play and critical thinking becomes possible” [173]. 


1.7 Methodological Themes 

While aspeets of the above conceptions of roles are compelling, they do not fully 
characterize all aspeets of the media artist. And in fací, they contradict each 
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other in some fundamental ways. Of course everyone has múltiple roles, and 
some contradiction provides necessary tensión. How then can the media artist be 
characterized, what practical methodologies could embrace these tensions? 

A host of artists and researchers llave attempted to offer methodologies that 
incorpórate arts practice and research. Many of these attempts are directed to 
media artists that have a studio-based practice, or that attempt to work witli 
technologists. That is, their methodology follows from the role they imagine for 
the media artist. These methodologies are largely geared toward art production. 
Some attempt to frame research methodologies in terms of theory related to cul¬ 
tural practice and art history. However, the methodology of media arts (as I 


am describing it) is increasingly drawn from pragmatic concerns. Practice-based 
research can be defined simply as “a research methodology in which an artifact 
is created by the researcher.” Mick Winter, an English composer, explains its 
differences from ordinary research methodologies, noting that it does not “ana- 
lyze artifacts or texts created by someone else“ or “directly study an individual 
or group of peoplc.” Rather, practice-based research has the interesting property 
where the thing that is stuclied “does not exist until the research project begins 
to c reate it” [231] , 

ümest Edmonds and Linda Candy, through a series of case studies with~pní 
icing artists, examine the “research trajectory” of media artists. They obser 
that practice- ba¡ 

these “research elemente” are often done in a non-linear way. Media artists are 
concerned witli creating artifacts, installations, or performance; the process of cre¬ 
ating and exhibiting is both an expression of a “conceptual structure” and also a 
mechanism that can lead the artist to formúlate or reformulate theories about the 
contení and meaning of his or her work. They find that artists do not normally 
rigorously evalúate their work in a formal way, but rather critique it intermittently 
during the creation process [55]. 

Estelle Barrett, an Australian proponent of practice-based methodologies, lists 
lational questions that need to be asked when explamillg'Cíea 
research: “What is it? Why do we need it? What can it do within the context of 

How do we judge its success and valúe 
low artistic production could be “validated” as a usetul reseaT 
amsm: ”What did the studio process reveal that could not have been revealed 
bymrTTrHtPr mndr nf nnqnirv?” She offers the idea of c rjtjml intrrprH~TtÍry¡Tnr 
“exigesis” as a way to tliink about artworks, and in particular in using Creative 
arts as a way to demónstrate processes rather than resulte [10]. The importance 
of Creative research stems from the “growing recognition of the limite of tradi- 
tional ways of representing the world” [59] and the awareness that it can lead 
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to “new ways of representing ideas and of illuminating the world and domains of 
knowledge.” 

These methodological approaches aim to articnlate and codify some of the 
conunon mechanisms artists use to produce a stronger and more meaningful body 
of work. However, they are largely focused on the creation of the artwork, and 
not integration with other forms of research. Other artists and scholars llave 
more specifically examined methods that artists use to further research that is 



a series of projects. She writes: “In isolation, scientific and artistic processes 
pursue orthogonal ‘directions’ of creativity, but, in combination, they allow their 
participant to access new areas of ideas, imagination, and innovation” [ 86 ]. Sirni- 
larly, Andrew Brown and Andrew Sorensen, live coders who perform in the dúo, 
aa-cell , examine their process of developing algorithmic multimedia art to explore 
“the role of aesthetics in research” and to demónstrate a methodology whereby 
“knowledge is created and expressed through a conversation between research and 
practice.” They find that their practice-based research draws on experimental 
and conceptual traditions of theory. They use their art practice as a system for 
making hypotheses and empirically judging outcomes; but they also, in part, see 
their work as logical, non-empirical research that relies on “measures of internal 
consistency.” At the same time, they emphasize the importance of turning this 
personally consistent knowledge into shared knowledge, transforming the knowl¬ 
edge of their craft through its expression “as a method, process or theory.” In 
particular, they note the symbiotic relationship between creating art and creat- 
ing software. In addition to using software to make their art, their art inspires 
how they structure their software. Both processes are iterative and involve rnany 
“refactoring” steps. They likcn their art performances to the release of a new 
versión of software. They consider their research contribution to be in the field of 
applied Computer Science; their practice contributes to discover or create the best 




action designers and researchers, Elisa Giccardi and Gerhard Fischer. 151 their 


O 


time, evolving as problems and needs change. Although they discuss in particular 
the design methodology for creating new artworks and new software tools, some 
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of their ideas are applicable to collaborative research in general. In particular, 
they note that the involvement of múltiple perspectives, the need for iterative de- 
velopment, and, in particular, the fact that many problems are “ill-posed” require 
flexible new ways of working and thinking about work. For them, “improvisation 
and evolution are more than a luxury: they are a necessity.” That is, when So¬ 
lutions cannot be specified in advance and when there may not even be a clear 
way to frame these problem, it may not be possible to delinéate an end goal for 
the collaborative project. Instead, they provide some ideas about how to take 
advantage of a collaborative design process that necessarily needs to evolve over 
time. The reformulated goal is not to create a specific outeome, but rather to 
define and create “social and technical infrastructures in which new forms of col¬ 
laborative design take place.” In some sense, the goal of the collaborative design 
is to design good collaborative design Systems! In other words, there is an el- 
ement of commnxul^-kwlding oí orgamzational ps_ycholog_y i«—tdieii__work. They 
alsodescfíbe specific methods that encourage effective collaboration. For ins 
íey discuss the idea of “externalization,” which means the creation of a tool or 
systern that encourages or forces participants to externalize their understanding 
of the project and each other’s roles. By so doing, participants are forced to ex- 
their understanding of the material and the research process in general. If 
it becomgs-eteajij Tat t here are misunderstandings or differing perspec tfvesrTnen 
this process exposes it/iclecilly leadmg tu uiuie eífeetive communication. Another 
idea rlicnnc p j g tlm i n tr^Hn n ti n n "f nn n cnmpnn ent of th e open 

5mTMediators are emergent languages that emerge through the process o: 
iboration. Because problems may not be well-defined and may evolve over time 
it nr^v~h n3 ~4i#iciilt d Q s^r ibe them at the outset of the pro j^A 
concept of externalization, they discuss how tools could be built to encourages 
the participants to share their knowledge and to create language or codes that 
describe their new ideas. 

In large part, these practice-based methodologies are aimed at introducing em- 
pirical research concepts, such as formal evaluation mechanisms, into art practice, 
both in order to validate the intellectual contribution of the artwork and also to 
help the artist describe and clarify their aesthetic concepts. A secondary aspect 
of this work is to think about how iterative, production-oriented, and “bricolage” 
techniques might be useful addition to interdisciplinary research projeets. 


O 


1.7.1 Emerging Methodologies 

Based upon the survey of roles and methodological approaches described above, 

I present four methodological themes can be used to more accurately be used 
to define and guide the activities of media artists in art-science contexts. These 


. 
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Theme 

Source 

Augmentation 

Shneiderman, Creativity Support Tools [195] 

Card, Sensemakinq / Information Foraging [26] 

Kramer, Auditory Displays [118] 

Jennings, Narrative Structures for New Media [105] 
Kuchera-Morin and Pcliti, Hydrogen Atom Project [177] 
Sutherland, Sketchpad [205] 

Wattenberg, Shape of Song [226] 

Electronic Visualizaron Laboratory, CAVE [43] 

Provocation 

Kac, Alba [108] 

Raley, Tactical Media [173] 

Critical Art Ensemble, Free Range Grain [42] 

Cox, Coding as Aesthetic and Political Expression [39] 

Harris, We Feel Fine [110] 

Shanken, Art in the Information Age [191] 

Mediation 

Lovejoy, et al., Context Providers [135] 

Legrady, Collaborative Research and Education [126] 

Vesna, Toward a Third Culture [217] 

Scott, Artists in Labs [187] 

Victor, Media for Thinking the Unthinkable [220] 

Generation 

Fishwick, Aesthetic Computing [64] 

Interval Research Corporation [131] 

Turkle, Bricolage Programming [212] 

Plautz, New ideas emerge when collaboration occurs [170] 
McLean, Live Coding Programming [147] 

Wilson, Elaboration on the Approach of Art as Research, [229] 
Xerox PARC, PARC Artist-in-Residence [97] 

Bell Labs/Explorations in Art and Technology [24] 


Tablc 1.1: Example texts and artworks associated with the different methodolog- 
ical themes. 
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themes will be used to ground the contributions of the projects I describe in 
later chapters. Table 1.1 catalogs which texts and artworks are most related 
to these themes, providing a link between various media arts activities and my 
methodological themes. Although, as I have discussed, there is a wide range of 
thinking about these activities and various, sometimes conflicting terminologies 
to describe them, these themes synthesize important aspects of these activities. 
Table 1.2 lists tasks associated with each of these themes. 


1.7.2 Augmentation 

The theme of augmentation is centered around the idea that certain media arts 
projects are effective at enhancing important aspects of research in order to make 
more accessible and more effective, both for outsiders and to domain-experts. 
That is, a media arts project is a methodological approach to understanding data. 
A fundamental activity of art is thinking about representation. Research into 
visualization and sonification are obvious areas in which media artists can make 
contributions. Similarly, thinking about immersion and interactivity as ways to 
augment research are also highly active research areas that make sense for media 
artists to explore. The creation of artworks, whether static or interactive, involves 
issues of engagement and how to use the “language of art” to tcll stories and 
provide context and narrative, and media artists could apply this know-how to 
make research both more accessible and more meaningful. Additionally, the tool- 
building interests of media artists can also serve to augment research, as media 
artists apply their creativity to novel ways of working with data representation and 
interaction. Ben Shneiderman’s interest in “creativity support tools”, as discussed 
above, clearly fits into this theme. Media arts projects, even if they have personal 
or cultural outputs, are already interested in augmenting research. Media arts 
projects will find interesting opportunities to augment existing research and also 
to provide interactions and representations that lielp researchers make insightful 
connections. Media artists can work in art-science contexts sirnply by recognizing 
that their work can function as augmentation. Sorne potential activities inelude: 
iterating over a specific, manageable data set rather than developing a universal 
technique; using media arts projects to explore research narratives, to help users 
turn raw data into a story; or analyzing the existing representation techniques (if 
any) to see what can be augmented or improved. 


1.7.3 Generation 

The theme of generation involves the idea that media arts projects can generate 
research topics. Simply by following an artistic inclination, a media artist will 
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Theme 

Task 

Generation 

Implement ideas without clear hypotheses about outeomes 
Clarify limitations/boundaries of knowledge and expertise 
Bring existing ideas into new contexts 

Augmentation 

Enhance data representation 

Enhance interaction with data 

Make data more accessible to domain experts 

Make data accessible to non-experts 

Show provenance of data 

Provocation 

Question assumptions 

Point out biases and blind spots 

Indicate cultural implications 

Expand audiences 

Mediation 

Create tools that make it is easy to represent concepts 
Create projects that facilitate communication 

Support knowledge-sharing 


Table 1.2: Potential tasks associated with the different methodological themes. 


quickly run up against personal and disciplinary limitations. Many times what 
the artist wants to do is not possible, or not accessible. Through a bottom-np 
approach, media arts projects can generate valid, interesting research questions. 
Moreover, the exploration of a particular area of interest can serve as a way to 
explicitly trace the contours and boundaries of the research, of precisely what is 
not known or is not easily accessible. In addition to surveying the research topic, 
media artists, who are often experts at particular engineering tasks or generating 
creativity support tools, can “switch hats” and turn from idea generation to actual 
research, whether it be in the area of augmentation, or other research. Of course, 
it is impossible to be an expert in everything, and this may be a point where the 
media artists now has a precise enough understanding of what they don’t know 
in order to seek out appropriate collaborators. Media arts projects will naturally 
come across interesting research topics and run into technical, representational, 
and even fundamental issues that can be explored. Sorne potential activities that 
can purposcfully leveraged in art-science contexts inelude: defining a project or 
series of projects purposefully to explore a particular area; keeping track of where 
Creative exploration becomes research. and being able to switch from exploratory 
to methodical approach; and actively looking for these boundaries instead of skirt- 
ing around them. 
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1.7.4 Provocation 

The theme of provocation involves the idea of challenges to the status quo, of 
introducing alternative perspectives and interpretations. Media arts activities 

llave Ol’iginality aS a gOal, f frjnlrinrr n^r, nrinn Uy r j ^nnlir^ni nf r q PQ 11r|1 p f q 

from different points of vie^H ^¥liy ar e the stated goals of a project imp ortant/L - 
Are there are other more interesting or usetui goals í Are the mettiods bemg used 
to achieve these goals ideal? While there is a critical aspect to provocation, it also 
has a generative function. The conceptualist artist and interdisciplinary researcher 
Roland Jones writes, “If artists and designers want to participate in reshaping 
the political, social, economic and cultural agendas, they will have to begin to 
think beyond the exhausted forms of critical belligerence and mere consciousness- 
raising” [107]. This generative function is of a different nature than the theme 
described above. It is questioning and experimenting goals and methods; in the 
previous we are trusting the goals of the project in order to find new avenues 
of research. Through provocation, media artists can introduce an interpretive 
element to research and can provide new formulations through exploring cultural 
implications embedded in research questions. Some potential activities inelude: 
pointing out areas that are underrepresented in research and exploring ways of 
including those areas; exploring meaning and implications, not just functionality; 
and explicitly discovering guiding metaphors and aesthetic modalities in order try 
to frame research using other metaphors and modalities. In other words, the goal 
of provocation leads initially to complication, but ultimately opens up dialog for 
more interesting and meaningful research. 


1.7.5 Mediation 

The theme of mediation involves the idea that media arts projeets can function to 
build a shared language. Media arts projeets often consider, or are often explicitly 
examining, mediations between different realms, bridging between different worlds 
that may not always come together, or exposing how they do come together. This 
ineludes starting or forcing a conversation between the everyday and the rarefied, 
tlie ji atu x al and l.l i e l lmna l. the pn hlic ami the pr i vate, int umsl. i lip p i i ijm ! ¿> - a.lmqs t 
by definition, are making new knowledge through combining different areas of 
_and expertise. by presenting new perspectives and interpretatk n 
are informed by múltiple disciplines or wavs of thinking. Thus it is important to 
create processes ajl^strategies that support effective knowledge-shaHng^ What 
if this were an explicugbal of a piuject, rather tlian a side effecCT rn order to 
function in collaborative contexts, Victoria Vesna writes, media artists “must learn 
the etiquette and language of various disciplines [...] Negotiating the gap between 
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the canon of rationality and the fluid poetic is ultimately the goal of artists who 
work with communication technologies [...] Bridging and synthesizing becomes 
the art.” Media arts projects already engage in this kind of negotiating, bridging, 
and synthesizing - an important yet often unarticulated focus of media arts - but 
this nright be a valuable way for inedia artists to explicitly position themselves 
as builders of “mediation support tools,” tools that can help mediate between 
different research communities in art-science collaborations. Some potential media 
arts activities inelude: creating projects to represent or externalize thinking and 
understanding; exploring ways to transíate concepts frorn one domain to another; 
and creating projects that facilitate communication. 

1.8 Research Outline 

As a step toward defining a practical methodology, the following chapters will 
explore some of rny own interdisciplinary projects that have helped me to think 
about the nature of media arts in relationship to art production and art-science 
collaboration. Chapter 2 looks at the Cell Tango [75] and Data Flow [127] projects, 
along with a series of smallcr projects called Information Poems, and examines the 
possiblc research contributions of these media arts pieces, examining in particular 
the role of data visualization research and data art visualization. Chapter 3 de¬ 
scribes the Fluid Autómata [73] and Annular Genealogy [76] artworks and discusses 
how artworks can generate research topics. Chapter 4 describes recent interdisci¬ 
plinary collaborations with domain scientists, the Natural Materials Browser [70] 
and Augmented Communication Tools [78], two interactive visualization projects 
explicitly drew upon these methodological thernes in order to more effectively 
promote research. Finally, in Chapter 5, I summarize the contributions of this 
dissertation and discuss a new conference session, the IEEE VIS Arts Program, 
within the larger IEEE VIS conference that facilitates discussion about the role 
of art in visualization research. 
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Chapter 2 

Emergent Organization: 
Aesthetic Data Visualization in 
the Cell Tango , Data Flow , &¿ 
Information Poems Projects 


Modern media is the new battlefteld for the competition between database 
and narrative. 

- from “Database as Symbolic Form,” Lev Manovich [142] 


2.1 OverView 

The previous chapter surveys roles of media artists in collaborative projects and 
presents four clusters of approaches- what I am calling methodological themes- 
that could aid artists when formulating, creating, contextualizing, or explaining 
their work, especially if they are explicitly concerned with working in art-science 
collaborations. In this chapter I examine art projects that, while not involved 
directly in an art-science collaboration, nonetheless also engage with the method¬ 
ological practices described by these themes. That is, although they have a solcly 
artistic output, they present activities that are useful in the context of interdis- 
ciplinary research. Moreover, I highlight central tenets of the scientific method 
(as it has become loosely codified in the mid-20th century and beyond). In so 
doing, I look at ways in which media arts methodologies could facilítate aspects 
of empirical Science - a goal of art-science projects and a rational for the inclusión 
of arts in STEM curriculum. I specifically delinéate a series of useful media arts 
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activities and further suggest how such projects could make explic-it contributions 
to various research pursuits. 

2.2 Collaborative Data Art Installations 

2.2.1 Info-aesthetics Methodologies 

Andrea Lan and Andrew Vande Moere, in their article, “Towards a Model of 
Information Aesthetics in Information Visualization,” introduce a domain model 
that positions visualization projects on along continuums of focus: data, inter- 
action, and aesthetics [124], In their view, work that emphasizes aesthetics has 
an “interpretive,” rather than a “direct” purpose. However, they bclieve that 
successful data visualization projects could incorpórate both of these outlooks si- 
multaneously. They stress that it could be more useful to think about the term 
“aesthetics” as involving reasoning about context instead of limited to the realm of 
representation (as often happens in the visualization research literature). Though 
they discuss the different focus of art projects and research project, noting that 
art visualizations engage with culture, they do not explore specifically how these 
projects may be useful beyond the scope of media arts. In fact, whilc it may 
be true that the art projects on Lau and Moere’s spectrum of visualizations are 
not directly engaged in information visualization research agendas, I contend that 
data art projects nonetheless do have the potential to contribute to research, in 
addition to their cultural contributions. 

In the sections below, I examine two collaborative artworks that I helped to 
create whilc working with George Legrady Studio. These installations, Cell Tango 
and Data Flow, are projects that are based upon the interactive visualization of 
specific data sets. Specifically, I examine how these artworks are perceived and 
utilized in order to discuss sorne of the methodological thernes described in the pre- 
vious chapter. Although the projects are not art-science projects, per se, they are 
nonetheless of an interdisciplinary nature, and involve elements that are usually 
not incorporated into art projects. I show that, in various ways, these artworks 
can be uscfully explored in terrns of the methodological thernes, augmentation, 
mediation , and generation. These thernes can be used to frame the potential con- 
tribution of media arts projects and to characterize a possible methodology for 
the creation of projects that seek to have a cultural impact and to engage with 
research agendas. 
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2.2.2 Cell Tango 

Cell Tango is an interactive installation that has been presented in fine arts mu- 
seums, in galleries, and at special events. 1 The pnblic visiting the exhibition is 
invited to interactively contribute to the project through submitting personal cell- 
phone irnages to the installation [75]. These images becorne the primary content 
source of the Cell Tango artwork, and are stored Online at the Flickr photo man- 
agement and hosting website 2 . The gallery display consists of a large cinematic 
projection that presents the contributed cell phone images in four animation con- 
figurations - ‘Cell Bin,’ ‘Cell Clusters’, ‘CellBurst,’ and ‘Cell Finale.’ The 
public is asked to add descriptive tags in the subject heading along with each 
submitted image. These descriptive tags function as search queries for retrieving 
other images from the larger Flickr database of public photography and also as 
organizational devices to create thematic clusters of images. The intent is to ex¬ 
plore the potential of unexpected juxtapositions where common semantic labels 
can generate interesting visual relationships. 

The animation ‘Cell Bin’ consists of visualizing the most recent images in the 
database on the screen. We developed a custom bin-packing algorithm that selec- 
tively places large images first and then gradually filis in blank spaces with gradu- 
ally smaller images until all of the empty screen spaces are filled. ‘Cell Clusters 5 
consists of thematic clusters of found images that are related to an image submit¬ 
ted by a participant. The photos in the thematic clusters themselves are generated 
by an analysis of the contributors’ tags, and are placed around the contributed 
image (which is differentiated by being given a yellow frame). The ‘Cell Burst’ 
animation throws images on the screen that then open likc bursting fireworks, 
placing the tags once the image appears, followed by the found Flickr images 
associated with each tag. ‘Cell Finale’ concludes the visualization sequence by 
rapidly placing on screen all of the contributed images in the database one af- 

1 Cell Tango premiered at the International Society of Electronic Arts (ISEA 2006) annual 
conference exhibition in San José, California in June 2006 under the title “Global Collaborative 
Visual Mapping Archive.” Since then, iterations of the projects have been included at many 
venues, including: George Legrady’s solo exhibition at the Pari Nadimi Gallery, Toronto (2006); 
“Speculative Data and the Creative Imaginary,” curatecl by Pamela Jennings, National Academy 
of Sciences Rotunda Gallery, Washington, DC, (2007); Inauguration of the National Theatre 
Poitiers, organized by Hubertus von Amelunxen, Poitiers, France (2008); “Cell Tango,” curated 
by Sarah Smarch, Ford Gallery, Eastern Michigan University, Ypsilanti (2008-2009); “Scalable 
Relations,” at the Beall Center for Art & Technology, UC Irvine (2009), curated by Christiane 
Paul; and “Cell Tango,” at Davis Museum and Cultural Center, Wellesley College, Wellesley 
(2009), curated by Jim Olson. Sonification was added and premiered at the Lawrence Hall of 
Science, UC Berkeley (May 2010) and then showcased at the Poznan Biennale, Poland (Fall 
2010 ). 

2 http://www. flickr.com 


33 



Chapter 2. Emergent Organization 



Figure 2.1: A photo of an earlicr iteration of one of the Cell Tango visualizations. 


ter the other in the spectacle action of fireworks exploding in the sky. Figure 
2.1 shows an early iteration of the Cell Tango project, Figure 2.2 shows an ex- 
ample of the ‘Cell Burst’ visualization, and Figure 2.3 shows an example of the 
‘Cell Filíale’ visualization. 

The artwork is dynamic; the arrangement of featured photographs is continu- 
ously being generated in real-time according to the rule-sets defined by the custom 
software. The software development involves significant engineering and problem 
solving, but it is driven by cultural concepts and aesthetics. In other words, it 
was designed with a “creative pulí” as opposed to “technological push.” Works of 
art tend to coalesce concerns, goals, and meaning, all of which evolve at múltiple 
stages in the work’s evolution. Tírese stages inelude the initial planning and design 
phase and the continued iterations of transforming the design during production. 
Cell Tango addresses a variety of concerns. It is an artwork that is about the 
creation of an archive of images, in this case limited to cell phone images. The 
project is based on the participatory, public contribution of data. Each exhibi- 
tion represents a collection that is the sum of all contributions niade by the public 
during the length of the exhibition. So at the start of an exhibition, the collection 
is empty; and at the end, the collection is closed. Each time the project is exhib- 
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ited, the sum of all images collected during the exhibition’s lifetime becomes the 
embodied visual archive specific to that event. 3 

Relevant research literature related to Cell Tango’s foci of interests ineludes 
Tim Kindberg’s discussion on the ubiquitous transformative role of cell phone 
camera usage [114]. Kirsti Lehtimaki and Taina Rajanti discuss the increased use 
of picture taking frorn specialized occasions to the documentation of ordinary life 
[129]. Andrew Miller and Keith Edwards address the social network aspect of 
irnage storage sites, including Flickr [151]. Nancy Van House articulates a priori- 
tization for irnage sharing as story telling, self-representation, self-expression, and 
documenting of everyday life over the long-term archival, collection opportunities 
that the photo management site offers [214], The artist Golan Levin has assem- 
bled a useful reference page of cell phone art projeets 4 , however rnost projeets 
use the cell phone as a sound instrument in group performance, or else as control 
devices by which to activate events in site-specific locations, for example by turn- 
ing on architectural fagades with embedded LEDs [102], The media artist Jonah 
Bruckner created an early cell phone image-based work titled “Phonetic Faces” 
that consists of a mixed collage of images and portraits on a display contributed 
by the public- present at the exhibition. This work used the cell phone to activate 
and control an onsite video camera that did the visual recording to a Computer 
that assembled the images 5 . 

The Cell Tango project may be understood as an exploration of the process of 
constructing and conveying meaning through the organization of visual elements 
(the images) according to a set of rules or conventions defined in the algorithms 
(the syntax) used to create the visualization. This positions the project within the 
context of a structuralist, cinematic tradition, as defined by the French film theo- 
rist Christian Metz in his analysis of semiotics and film [149]. Cell Tango follows 
the structuralist film theory model of constructing meaning through juxtaposi- 
tion, adding an uncertainty component as the aleatory selection of accompanying 
images are delivered according to comrnon tag labels. The outeome of the selec¬ 
tion covers a continuum from a puré literal, analogous matching visual content 
to visual content that may be totally unrelated at the literal, similarity lcvel, 
but which may have implicit metaphoric or associative relevance. For instance, 
a tag that says “circular” might result in an analogous irnage of a tire, whereas 

3 Cell Tango has a prior history that extends back to its initial development by George Legrady 
via a 2004 Cañada Council grant. The intent of Cell Tango was to create follow up to the Pockets 
Full of Memories project. Pockets Full of Memories involved a heavy and cumbersome data entry 
station, and a main goal of Cell Tango was to make the data input station more portable and 
to give global rather than only gallery-specific local input access. 

4 http: / / www.flong.com/texts /lists/mobile_phone 

5 http: / / www.coin-operated.com/coinop29 /2010/05/01/phonetic-faces-2003/ 
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another image which is tagged less literally with words like “yummy” or “sweet” 
might result in images of ccll phones, cupcakes, chocolate drippings, images of a 
mug, a dooclle, et cetera. One of Structuralism’s fundamental principies is that 
underlying structures lie beneath the surface or the appearance of things [8]. In 
Cell Tango, the algorithmic processes express the projects’ structure through its 
programmatic rules, and this functions as the binding by which the juxtaposed im¬ 
ages in Cell Tango are organized. When the meanings between juxtaposed images 
are evident, the viewers’ experience in fact becomes less interesting than when 
the meanings are harder to interpret. Hurnans naturally attempt to create and 
interpret meanings out of any information Corning their way, and the less evident 
pairings turn out to be the rnost stimulating in the viewing experience. 



Figure 2.2: A detail from the ‘Cell Burst’ visualization. 


Cell Tango features the interplay between two image sets: the “known” System 
of the visiting public’s ccll phone contributions; and the “unknown” open System 
of tag retrieved associative images from the larger Flickr archive. At the start of 
the exhibition, the collection is limited to a few seed images, but then builds up 
throughout the length of the exhibition. When the exhibition is completed, the 
collection culminates into the unique record representative of that installation’s 
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specific set of conditions, visitors, and cultural and temporal contexts. Following 
the 2006 premier at the International Society of Electronic Arts in San José (un- 
der a different title, “Global Collaborative Visual Mapping Archive”), Ccll Tango 
has been featured in a variety of venues that inelude: an opening event at a na- 
tional theater in France; a month in a commercial gallery; a long-running media 
arts exhibition at a West Coast research university gallery; a two month exhibi- 
tion at a public Midwestern university; and a four month exhibition at an East 
Coast private university. While Cell Tango functions primarily as an artwork, 
it is also an investigation in the study of how the functionality of photographic 
image-making is changing through cell phones. Each exhibition is a data collec- 
tion environment as the public submits images, which are then stored for later 
analysis. Each venue has its own socio-economic/cultural context with sorne no- 
ticeable differences. User responses revealed that images were expressive of local 
community interests, social perspectives, and varying degrees of implementation 
of the functionality of photographing, from recording special events, to recorcling 
phenomena, or idiosyncratic expressions. Additionally, there is interplay between 
the collection of submissions through the artwork’s context and the public images 
retrieved from Flickr through the associated tags submitted by users. 

The artistic requirements of Ccll Tango necessitated the creation of a robust 
software implementation that allowed effective user interaction via the ccll phone, 
animation of the visual elements within the different scenes, and the extensibility 
and adaptability to add new scenes and to easily relocate the project in different 
environments. Cell Tango visualizes an evolving database of cell phone photogra- 
phy and it is itself an evolving project that changes according to the venue, but 
also in order to explore new ideas and representations. The System is separated 
into four processes that run simultaneously: the data gatherer, the visualization 
modules, the local network management component, and the graphics renderer. 
Communication between these processes occurs through writing and reading from 
concurrent data structures, which efficiently and safely allow múltiple threads to 
access and update the same information asynchronously. Because the main user 
interaction with the System is done via the submission of ccll phone photography, 
a primary technical goal of the System is to retrieve new photos as soon as they 
are available. Another goal of the project is to explore the relationship between 
the user-submitted photos and the rclated public photos. In order to facilitate 
the rapid retrieval of new photos and the exploration of related photos, a data- 
gathering component is placed in a background thread that continually checks the 
Flickr database. The process gathers both user-submitted photos and tags, as well 
as related photos from the public Flickr photo pool that are related to the user- 
submitted photos via their folksonomic tags. A local data store is kept in memory 
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that holds the most recent user-submitted photos and wcll as selection of photos 
selected from the pool of other, less recent, user photos. Other important aspects 
of the software design included: the development of a real-time bin-packing al- 
gorithm that worked with streaming data; concurrent data structures that could 
rapidly store and retrieve large collections of user photos, tags, and related photos; 
and general robustness that allowed the project to be run for the duration of the 
installation (in some instances up to six months) without human intervention. 



Figure 2.3: A participant standing in front of the Cell Tango installation. 


The system uses a custom 3D graphics framework callcd “Behaviorism” [72] 
which loads each of the photos on to hardware-accelerated textures and also pro- 
vicies a scene graph and various animation and timing techniques that control the 
layout and narrative of the different visualization modules. Each of the visualiza- 
tion modules processes the data that has been placed within the local data store 
by the data gatherer. Visual elements based on this data are placed on the scene 
graph, and then rendered via a dedicated rendering thread at sixty frames per 
second. Animations of the movement, size, color and structure of visual elements, 
as wcll as the movement of the camera view, can be programmatically defined by 
adding behaviors to the timing graph. Tliese behaviors update the objects in the 
scene graph at the start of each loop of the rendering thread. In addition to using 
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hardware-accelerated textures for rendering the photos and text, each visualiza¬ 
ron module is able to use OpenGL for advanced graphics techniques, including 
binding to custom GLSL shader programs. Because the Ccll Tango project has 
liad to evolve and adapt to new environments, the software system is designed to 
make it easy to add new visualization modules that transform and visualize the 
photographs and tags in novel ways. Additionally, some iterations of Ccll Tango 
experimented with the sonification of the visualizations, and included a network- 
ing component for propagating visual or data events to another Computer that 
then transforms these events into algorithmically generated composition. 

The preceding description indicates an interesting dynamic: software engi- 
neering enables artistic goals, but at the same time the artistic goals require the 
development of new engineering methods. Media arts projects have research pos- 
sibilities beyond the cultural goals and the software engineering aspects. These 
possibilities are especially interesting as they emerge unexpectedly out of the 
project itself, and simultaneously involve Creative and technical thinking. These 
ideas can be grouped under the themes of generation, augmentation, and media- 
tion. While not fully explored in this project, I contend that media arts projects 
could be reframed to highlight these potential research avenues. That is, the cre- 
ation of the media arts project is in and of itself a methodological tool for creating 
and conducting research. 

2.2.2.1 Generation in Cell Tango 

Some components of Cell Tango only emerged through the process of trying out 
many iterations of different ideas related to the main theme of interacting with 
ccll phone photography. These components required further study, and presented 
interesting avenues of research beyond the installation itself. That is, the project 
itself was a generator for potential future research in unexpected domains. For 
example, a main aspect of the project involves searching an image database using 
an image and its associated folksonomic tags. The idea for searching databases 
using images was introduced at least as early as 1980 [32], Since then, much 
research has been done to improve the results of using images to find similar 
images. For example, in 1989 a technique was developed to encode only the most 
relevant aspects of an image for faster an more accurate image retrieval [125]. 
A similar technique is described in [58], except a sketch is used rather than an 
image to retrieve images. More recent work focuses on combining image analysis 
techniques and feature detections with contextual clues and image metadata in 
order to produce more results more relevant to a particular query [239]. In 2013, 
Googlc Images introduced the ability to drag and drop an image and inelude 
semantic tags and nearly instantly retrieve accurate results [91]. Although at 
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the time the project was not informed by the literature in this research area, 
we nonetheless explored interesting approaches to image retrieval. Some of our 
ideas included: culling the results from Flickr by identifying simple features and 
comparing them to our input image; using features from connected photos in order 
to find images relevant to not only the original image, but all of the neighbor 
images in the installation; making use of múltiple tags to whittle down results 
into a manageable number. 

Another emergent idea from Cell Tango is the concept of presenting search 
results in a non-linear manner. Nearly all search interfaces impose an order on 
their results and visualize them in a list. In one sense, all the visualizations of Cell 
Tango opérate to present search results in ways that challenge this representation. 
Moreover, each visualizaron tries to contextualize the results in terms of the other 
images, in terms of previous or concurrent searches, an in terms of the installa¬ 
tion as a whole. The cycling quality of our System brings in new results and new 
rankings. In so doing, Cell Tango provides interesting juxtapositions that forces 
viewers to think about the narrative implications of search, something which is 
explicitly hidden in most search applications, or is governed by less meaningful 
metrics, such as popularity or relevance to advertising interests. Again, there is 
much research about these topics that we clid not investígate at the time, but that 
the project could potentially have been used to explore. For example, [95] exam¬ 
ines the complex dynamics of collaborative tagging, and [243] discusses methods 
to improve search results through topics diversification. 

It would be interesting to create an installation with the intention of pursuing 
researching the questions the research questions that emerge from the installation. 
Although this project did not directly engage with these research questions in a 
formal sense, it is clear that the project delineates their contours. Moreover, the 
artwork functioned not only as an aesthetic experience, but, in a sense, also as 
a data gathering system for conducting empirically-based experiments towards 
these topics. In Chapter 3, I discuss projects that switched between an artistic 
and a research focus, purposefully using the media arts project as a methodology 
for generating research topics. 

2.2.2.2 Mediation in Cell Tango 

In Chapter 1 I introduced the theme of mediation as a primary methodological 
focus for media artists. Cell Tango is explicitly engaged with thinking about how 
individuáis mediate between the public and private. Additionally, simply by virtue 
of the fact that it is both an art project and a complicated engineering undertak- 
ing, its creation involved the continual mediation between the Creative and the 
technical. While these mediations are not specifically applied to larger research 
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endeavors, they serve as examples of the type of focus that media arts projects 
could offer art-science discourse as well as interdisciplinary research in general. 
Mediation involves understanding and commnnicating fundamental differences in 
interpretation and perspective. Media arts projects are able to practically engage 
in this type of mediation, and at the same time, comment on them, experiment 
with them. 

Victoria Vesna’s book, “Database Aesthetics,” brings together various per- 
spectives regarding the cultural implications of quantifying and storing large 
amounts of data [218]. Christiane Paul, in her essay in that collection, “The 
Database as System and Cultural Form: Anatomies of Cultural Narratives,” ex- 
plains that “database aesthetics” projects explore ways of “revealing (visual) pat- 
terns of knowledge, belicfs, and social behavior.” She further explains that novel 
characteristics inherent in works of database aesthetics inelude the concept of mul- 
tiplicity, that is, enabling opportunities to view data frorn many perspectives and 
narratives. Paul explains that these types of projects are by nature provocative 
because they offer alternatives to normal or expected narrative structures. She 
cites the early net art project, Carnivore, (an open source packet sniffer created 
by Radical Software Group) as an example in which an artwork can, simply by ex- 
posing the normally hidden protocols and processes of sending information, make 
a “political statement about access and its control.” In so doing, it constitutes a 
shift toward a “networked approach to gathering and creating knowledge about 
cultural specifics” [165]. 

The Ccll Tango project explicitly investigates the cultural meaning and narra¬ 
tive effeets of translating personal photographs into the various visual representa- 
tions of the database of the installation. This is similar to an earlier project by 
Legrady, Pockets Full Of Memories, in which “the function of semantic interpreta¬ 
tion” related to private objeets scanned into a public installation [128]. A relevant 
recent essay by media theorist Nick Couldry talks about the difference between 
“mediation” and “mediatization,” where mediatization indicates a transfer of an 
item of personal or cultural significance into a publicly visible social network [38]. 
Another way to think about the Ccll Tango project is that it provides a space for 
participants to reflect upon the relationship between their private irnages and the 
public interpretation of the image as it becomes mediatized. Issues between pub¬ 
lic and private are perennial, especially involving digital Communications industry 
[89], and much research investigates some of the potential negative issues asso- 
ciated with the loss of privacy associated with mediatization. Less polemically, 
Leah Lievrouw, a co mmuni catión theorist, writes about the importance of “resit- 
uating mediation at the center of communication” in order to “help us conceive 
of communicative action, social context, and material resources as inextricable, 
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co-determining aspects of sociality, interaction, expression, meaning, and culture” 
[132], In her article, ”Between Archive and Participation: Public Memory in a 
Digital Age,” The rhetorics scholar Ekaterina Haskins looks at the September 11 
Digital Archive to explore the nature of dynamic archives in the age of social net- 
works [98]. Although she notes the tensión between storage/ordering functionality 
and presence/interactivity functionality, she proposes shared, dynamic archives as 
an effective way to store personal meanings, and superior in some respects to the 
“hegemonic official memory” of closed archives. 

2.2.3 Data Flow 

The Data Flow project is a site-specific installation commissioned by Gensler De- 
sign for the Corporate Executive Board Headquarters. Corporate Executive Board 
(CEB) provides research and analysis of various business sectors to over 16,000 
paying members (rnostly business executives) throughout the world 6 . Data Flow 
consists of three visualizations that map the members’ daily interactions with 
the CEB web portal. These visualizations are situated on the “Feature Wall,” a 
three-story installation space adjoining an open staircase on the top floors of the 
CEB headquarter offices in Arlington, Virginia. Each of the three visualizations 
of Data Flow consists of three horizontally-linked screens displaying hi-resolution 
animations. A main server retrieves and analyzes a constant flow of user data, 
including search terrns that are used to find documents, information related to 
the webpages the user visits, and geospatial data indicates where and when the 
user logs in to the CEB web portal. Data relevant to each visualization is then 
forwarded to a sepárate Computer that handles the layout and animation of a 
particular visual representation of the data. One visualization displays statistics 
about which webpages are most active at a given time. A second visualization 
displays real-time information regarding the location where members are logging 
in from. The final visualization displays an animated record of the search terrns 
used to access business documents. Figure 2.4 shows two of the Data Flow visual¬ 
izations over two floors of the CEB headquarters. Figure 2.5 shows a detail from 
one panel of the GeoMap visualization. 

2.2.3.1 Bigram Visualization 

The “Bigram Visualization” animation displays a constant stream of the most 
common keywords culled from members’ web queries over the last hour. Each 
keyword is presented in tándem with thcir most popular associated words, forming 
a series of bigrams. Colored lines show the relationship between the bigrams and 

6 http://www. executiveboard.com/exbd/about 
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what type of page (which “sector”) within the CEB portal the user ended up at 
via the search terrns, as wcll other information about the member’s level of access 
(their “program”). Tliis visualization makes visible the members’ search interests 
as they investígate various sectors in the web portal. The animation begins by 
grabbing the most recent 20000 events within the last hour and analyzes them 
for the most popular 12 keywords (sterns). Each stem is linked with its with 
four most commonly associated words to create bigrams. The bigrams are then 
placed on screen, moving from right to left. Bigrams appear on screen and travel 
diagonally upwards or downwards, exiting at their opposite córner. Each bigram 
is linked to its various sectors, appearing above it, and its program, appearing 
below. Bigrams accelerate as they move across the screen, and the text boxes 
shrink as they move from right to left. A vertical line moves across the screen 
separating stem groups. 



Figure 2.4: The Bigram visualization and the GeoMap visualization of the Data 
Flow installation. 

While bigrams are often used as a way to conduct certain types of linguistic 
analyses, they, at the time, were not usually used for providing insight into more 
focused data sets. Over the last few years, and especially with the introduction of 
the Google Books Ngram Viewer [167] it has become more conmion to use bigrams 
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to analyze various types of text. For instance, in NGram Viewer, users can search 
for particular terms over specific types of texts or time periods. Data Flow uses 
bigrams to provide a window into the main interests of users of the CEB website. 
Moreover, because the installation perpetually analyzes new, incoming streams of 
data, it serves as a way to track how these interests change over time. 

2.2.3.2 GeoMap Visualization &c Coil Map Algorithm 

The “GeoMap Visualization” shows temporal changes within a geographical rep- 
resentation. Data representing the number of elements related to user interaction 
on the CEB website is stored within the data graph, such as the geographical 
location of the user, information about the type of account they have, the search 
terms that were used to find particular documents, and what section of the CEB 
website they are searching. New data is retrieved every 5 minutes from a central- 
ized SQL database and outdated data is purged once it is an hour oíd. At the 
start of each iteration of the geographical visualization a sample of the searches 
from the last hour is collected. The visualization begins with a map of the world 
broken into 2048 * 3 cells. Each of the search samples are processed in order and 
the cell containing the location expands distorting the world map slightly. After 
a number of samples has have been processed it is easy to tcll which parts of the 
world have active users. After all the samples are processed, each of the behaviors 
used to change the visual appearance are “rewound” at a faster rate, and the map 
“unwinds” back to its original state. Because each of the visual behaviors runs 
asynchronously and because the speed of the cell expansión animation may be 
slower than the rate of processing new samples, the overall effect is that the map 
changes in a fluid manner. 

The visualization uses an information visualization rnethod called “Coil Maps,” 
an animated treemap algorithm created for this project, to display a dynamic 
overview of geographic data [68, 72], A world map is initially recursively subdi- 
vided to a specified depth. The final subdivisión constitutes the lcaf tiles, which 
cover the entire map. As new events are attached to a leaf tile, the tile increases 
in size, which causes its parent tile to increase in size, which in turn causes its par- 
ent tile to increase in size, and so on up until the root subdivisión. This causes a 
change to be very apparent on the local level, but also to more subtly show changes 
on a global scale. For examplc, if initial tiling covered a map of the world, and 
an event arrives on, say, New York City, then that tile would immediately greatly 
increase in comparison to its neighboring tile, but also lesser changes would affect 
all of the parent tiles, up to and including a minor increase in size of the northern 
hemisphere and a related minor decrease in the size of the Southern hemisphere. 
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Figure 2.5: A detail of the GeoMap visualization of the Data Flow installation. 


Tliis distortion allows a viewer to see at a glance the most active regions of a map 
over time while retaining the perceptual grounding of the map itself. 

The use of cartograms that distort geographical features in order to emphasize 
particular data relevant to a visual presentation is a rnuch studied and ongo- 
ing topic. The cartographer Waldo Tobler has examined the history of digital 
cartograms beginning in the 1960s [208]. Advances in methods for generating 
cartograms continué to be developed. Recent work by a number of different 
researchers explored various tradeoffs between different aspects of cartogram al- 
gorithms, including issues related to readability, optimal layouts, and flexibility 
[60, 215, 235]. An article by Marc Van Kreveld and Bettina Speckmann looks 
specifically at ways to animate rectangular cartograms (using a different method 
than the Coil Maps algorithm) [60], and a recent survey of geovisual analytics 
by Gennady Andrienko, et ah, notes that the simultaneous representation of tem¬ 
poral and spatial aspects of data continúes to be an important area of research 
[4]. Daniel Keim, et al., has also cataloged the current scope and challenges in 
the field of visual analytics, identifying the representation of dynamic, ternpo- 
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ral, geographically-situated data as one contemporary issue that requires further 
research. 

2.3 Information Poems 

Here I introduce a series of smaller data visualization projects I cali “Information 
Poems.” Tliese smaller, individually produced, artworks use raw data in a very 
different way, and are example illustrations of the therne of provocation. Although 
not a direct collaboration with a single person, these projects are unwittingly 
collaborative in a sense since they rely on data generated by others. I cali these 
pieces “poems” since they are lightweight pieces: they take less time to create, 
they make a smaller statement, and they are less substantial in many ways than 
full art installations. Moreover, they have a “lyric” sensibility - They are evocative 
and provocative, but do not attempt to engage with the complete, or even the 
main, ramifications of the data sets they utilize. 

2.3.1 Infrequent Crimes 



Figure 2.6: A screenshot showing one of the crimes frorn Infrequent Crimes. 

Infrequent Crimes is a data visualization art piece that was shown originally in 
Santa Barbara and then again in San Francisco.' In both cases, the main médium 

7 Infrequent Crimes was presented in “Spread: California Conceptualism, Then and Now,” 
at SOMArts Gallery, San Francisco, California, 2011. Curated by OFF Space (Kathrine Worel, 
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of the piece was a dataset of crimes reported to the pólice and kept in a large 
database. In addition to the type of crime that was reported, information about 
the location of the crime was kept as well, recorded in longitude and latitude. 
While data visualizations incorporating crime data are popular (see for instance 
the New York Times’ Homicide Map 8 and Stamen Design’s Crimespotting 9 ), it 
is less connnon to visualize unusual minor crimes within neighborhoods. While 
there might be, say, an average of twenty crimes involving cars being broken 
into per night, other crimes are less frequent. For instance, the crime labelled 
as ”defrauded innkeeper” was reported only nineteen times in a single year in 
San Francisco in 2011. Similarly, a crime labelled as ”cruel children” was only 
reported four times. Giving a representation to these infrequent crimes provided 
an alternative perspective on the life of a city, as seen through the lens of pólice 
activity. The crimes were visualized in an easy to read manner. The ñame of the 
crime would pop up, along with the number of times the crime occurred in the 
last year. Then a number of tiles would appear, counting out the total number of 
crimes, where each tile represented one crime. Each tile contained either a) a map 
indicated where the crime occurred, or b) a photo taken from Google Maps Street 
View at that location. The result was a strange mix of the eerie and the humorous, 
where odd-sounding crimes, like “gunshot fired indoors,” or “bus mischief,” were 
placed against unpopulated views of Street corners, houses, and parks, providing 
an ocldly unsettling tour through locations of the city via the location of infrequent 
crimes. By so doing, I focused less on the crimes themselves, and more instead 
on the way the crimes are classified. That is, it serves to expose the arbitrary 
parceling of deviant behavior and the Byzantine penal structures that regúlate it. 
Figure 2.6 shows a screenshot from Infrequent Crimes. 

2.3.2 Turbulent World 

Turbulent World displays an animated atlas that changes in response to the in- 
creased deviation in world temperature over the next century. 10 The changes are 
represented by visual eddies, vórtices, and quakes that distort the original map. 
Additionally, the projected temperatures are themselves shown across the world, 

Elyse Hochstadt, and Emmanuelle Namont-Kouznetsov), and also in “Super Santa Barbara,” a 
group show at the Contemporary Arts Forum in Santa Barbara, California, 2011, curated by 
Warren Schultheis. 

8 lrttp://projects. nytimes.com/crime/homicides/map 

9 http: / / stamen.com/ projects/crimespotting 

10 Turbulent World was shown as part of “Brave New World,” at the Spare Change Artist 
Space, San Francisco, California, 2013. Curated by OFF Space (Kathrine Worel and Emmanuelle 
Namont-Kouznetsov). 


47 



Chapter 2. Emergent Organization 


increasing or decreasing in size to indicate the severity of the change. The data 
was generated by a sophisticated climate model that predicts the monthly vari- 
ation in surface air temperature across different regions of the world from 2013 
through 2099 [47]. The dataset is available at the National Climactic Data Center 
(NCDC), run by the National Oceanic and Atmospheric Administration (NOAA). 
More detailed information about CM2 Global Coupled Climate Models (and the 
data itself) can be found at the Geophysical Fluid Dynamics Laboratory (GFDL) 
website 11 . 

The goal of Turbulent World is to provide a window into projected data repre- 
senting current thinking by leading scientists about climate change. Scientific visu- 
alization often focuses on individual data samples. Visualizations that effectively 
capture large-scale Systems are more difhcult to represent. Additionally, climate 
change represents múltiple, intertwined Systems and necessitates new thinking 
about economics, policy-making, urban development, and other activities. Since 
modeling, representing, and explaining climate change is so challenging, Turbulent 
World doesn’t attempt to be a comprehensive visualization, but rather presents 
one aspect (the projected temperature) in order to provide a window into this com- 
plex system. An additional issue in representing climate change is that it exists at 
a scale that is hard to reason about, and thus, despite the major implications to 
our civilization, it is not something that peoplc think about very often. A recent 
article by Adam Córner argües that, by inventing new ways of representation, 
artistic projects keep environmental issues “in the public eye” 12 . Turbulent World 
has been shown at the Spare Change Artist Space in San Francisco, curated by 
OFF Space. It will also be shown in a planned exhibition in the Biosphere2, as 
part of the Climate, Adaptation, Science, & Art initiative. Figure 2.7 shows a 
detail from Turbulent World. 

2.3.3 Poetry Chains & Collocation Nets 

In their article “Deformance and Interpretation,” Lisa Samuels and Jerorne Mc- 
Gann examine a seemingly whimsical fragment written by Emily Dickinson: “Did 
yon ever read one of her Poerns backward, because the plunge from the front 
overturned you?” [48]. They choose to interpret it literally, asking how a reader 
could “release or expose the poem’s possibilities of mcaning” in order to explore 
the ways in which language is “an interactive médium.” Poerns, they clairn, citing 
Shelley, lose their “vital forcé when they succumb to familiarization.” They imag¬ 
ine instead unfamiliar deformations of literature that offer “a highly regulated 

11 http: / / nomads.gfdl.noaa.gov 

12 http://www. theguardian.com/sustainable-business/art-climate-change-communication 
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2067, September Average Deviation ¡n Global Temperatura: 1.36° 



Figure 2.7: Screenshot of Turbulent World displaying tire projected surface-air 
temperature throughout the world for September, 2067. 


method for disordering the sense of a text.” fn so doing, they posit that perfor- 
mative critical models provide an important “anti-theoretical” interpretation not 
available to traditional interpretative criticism [146]. 


winces with Jeliqht 
anonymous deliqh to know - 
as star: that drop anonymous 
till brooms fade softly into stars •• 
oh let me immediately íade 


Figure 2.8: An examplc of a Poetry Chain. When one poem is finished, a new one 
will start, using the last line of the finished poem as the first line of the next one. 

Two projects, Poetry Chains and Collocation Nets investigate the 1955 edition 
of Emily Dickinson’s complete poerns through various interactive animated nav- 
igations of collocated words. As such, they perform what Samuels and McGann 
term “experimental analyses.” Each of the visualizations displays a different pre- 
sentation of her work. Poetry Chains begins with two words and attempts to find 
a chain of words in a specified number of lines that connects them together, dis- 
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playing them as it succeeds. Collocation Nets begins with a single word centered 
in the middle of the screen. When the user selects the word, a random selection 
of its collocations pops out in a surrounding ring. Any of those words can be 
selected, which results in collocations of that word appearing. A user can toggle 
into an ambient mode of this visualization, which begins automatically cycling 
through all of the words, forever. 

These visualizations offer a continuously dynamic remapping of Dickinson’s 
work. The deformations present new opportunities for interpretation, sorne of 
which may lend themselves to successful insights, and others which rnight be lu- 
dicrous, or merely bland. Each of the visualizations performs this remapping in 
different ways. The Poetry Chain effectively runs a kind of smoothing operation, 
an averaging filter, by treating her entire corpus as a single poem. Additionally, 
it uses a depth-search algorithm to get between two points within the corpus, 
performing a non-linear “hopscotch” (with a poetic rather than narrative destabi- 
lization). The Collocation Net completcly disassembles the corpus into individual 
words and links them together, not grammatically, but instead by a frequency 
metric that corrclates words by the likclihood of their appearing together within 
the same line. Whilc it is unclear what exactly the interpretive valué of these 
remapping offers, it is interesting to think of them in relation to, or perhaps as 
a differentiation from, visualization projects utilizing the methods of information 
visualization or visual analytics. In those fields, it is assumed that the raw data is 
inhcrently atomic, and that the goal of the project is to enable users to recombine 
the data in different ways in order to facilitate new revealing and new interpre¬ 
tation, or what Peter Pirolli and Stuart Card term “knowledge crystallization” 
[168]. That is, they allow the user to create rnodels by the synthesis and analysis 
of data, through which hypotheses may be generated and then either validated 
or falsified. A recent articlc by Ben Shnciderman reframes the producís of infor¬ 
mation visualization projects as creativity support tools, where the goal of such a 
tool is to facilitate creativity: novel ideas and new perspectives. Poerns however, 
as noted in Samuels and McGann’s article, are not sirnply composed of irreducible 
raw data. Instead, the meaning in some sense is the raw data. But this meaning 
lives in the interaction between the text and the reader, and can not be extracted, 
simplified, summarized, or evaluated in any direct way. 

A primary task in traditional information visualization projects is to provide 
different kinds of overviews so that the user can, at a glance, get a general sense of 
the data before investigating aspects of the data in more detail. For textual data, 
which is unstructured, and which has more nuanced, complex meaning, it may 
be too reductive to attempt to provide this kind of overview. Instead, overviews 
of textual works are provided in different ways: through biographical informa- 
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tion, through a delineation of themes or topics, through summarization, or, most 
generally, by having the reader skim through the text. In lyric poetry however, 
where the themes are not easily explicable and no obvious narrative lends itself 
to summarization, overviews are necessarily interpretive acts. Through a series 
of ambicnt and interactive visual sketches, these two projects aim to provide the 
user with a loose overview of the language using a “skimming” metaphor, provid- 
ing perhaps ad-hoc interpretations, and allowing the user to investígate further. 
A second primary task of information visualization projects is to allow user the 
opportunity to find more detail about a particular element or set of clcments. A 
future task of these visualizations would be to allow users to the ability to view 
the poerns themselves when desired, and to bring up further information about 
a particular word as desired. That is, it is important for a user to be able to 
contextualize the investigations offered by the visualizations, and to have more 
control over the interpretive process. 

In devcloping these sketches, the first step common to all of thern was to 
parse the data into a format that enabled the visualizations. Each line of each 
poem is connected to all of the words in the line, and vice versa. And all of 
the words have a pointer to each word that it is collocated with, as well as how 
many times it is collocated with that word. Additionally, the total frequency of 
each word is recorded, as well the rank of the commonness of the word. Using 
only these data structures, stored in memory, was sufñcient for the Collocation 
Nets visualization. The Poetry Chain visualization required slightly more involved 
Processing. In order to find a connection between two words, the software performs 
a depth-first search, recursively scanning lines connected by collocations until a 
line containing the target word is found. Because this can be a time-consuming 
process, I limited the number search by removing every word and every line that 
the search encountered in order to dwindle the possiblc pathways it searched, 
and also to prevent infinite loops during the search. Because of this fairly crude 
filtering process, it is possiblc that a path will not be found. In this case I sirnply 
re-run the search. In general, a full path between the source word and the target 
word is found relatively quickly, usually on the first attempt. I also improve the 
chances of generating a successful path by reversing the source and target words 
if the target occurs lcss frequently than the source (and reverse the path when 
it is complete). Occasionally, when both the source word and target word are 
infrequent it may take more than fifty attempts to find a path, in which case the 
search sirnply gives up and the user is asked to try a different source and target. 
Figure 2.8 shows a screenshot frorn a Poetry Chain. Figure 2.9 shows two examplc 
screenshots from Collocation Nets. 
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In a recent article, Johanna Drucker writes that a book is “a snapshot, a 
momentary slice through a complex stream of many networked conversations, 
versions, and fields of debate and reference across a wide variety of times and 
places. A book is a temporary intervention in a living field of language, images, 
and ideas” [51]. That is, while the layout of book implies linearity, the underlying 
connections are complex and multi-faceted. By presenting the text in a non-linear 
manner, these networked conversations become more apparent, and invite new 
readings based on the reconfiguration of words. In “Speculative Computing: Aes- 
thetic Provocations in Humanities Computing,” Drucker writes that a goal of 
“humanities computing” is “to produce useful aesthetic provocations” that serve 
in part to emphasize that any data analysis is “constituted in an interpretation 
enacted by an interpreter” [52], In both the Poetry Chains and Collocation Nets 
projects (as well as the other “Information Poems”), I was interested in unortho- 
dox ways of representing the data in order to provoke a new attention to the 
meaning of the data, to highlight unusual or interesting components of the data, 
and to bring awareness to the structure of the raw data itself, whether it be crime 
data, environment modeling, or lines of poetry. 



2.4 Art as Research 

In this chapter, I introduced a series of data visualization projects that I created 
or was involved with, including two larger endeavors involving múltiple people 
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working with George Legrady Studio, as well as three smaller projects that made 
use of culturally interesting data sets. In so doing so, I examined different aspects 
of particular art activities that share something in common with research practices 
and that moreover could be used to enable research practice in other domains. 

In this concluding section, I draw parallels between artistic practice and sci- 
entific research. While there have been studies that have looked specifically at 
differences between practice-based and scientific research trajectories, for example, 
Mick Winter [231], and Ernest Edmonds and Linda Candy [55], I instead highlight 
the similarities, and show sorne opportunities for media art practice to enable sci¬ 
entific methodologies. Specifically, I look at at sorne of the different steps that 
have characterized the scientific method first dclineated in Francis Bacon’s 17th 
Century Novum Organum [9]. These steps inelude: observing and describing a 
phenomena; formulating a hypothesis; using the hypothesis to predict results; con- 
ducting experiments to test the results; and finally analyzing and interpreting the 
results [228]. 

2.4.1 Observing 

Warren Sack writes, in “Aesthetics of Information Visualization” about how visu- 
alization can be consiclered either “sublime” or “anti-sublime.” He takes issue with 
a statement by Lev Manovich, in which he discusses information visualization as a 
tool to transíate phenomena into “the scales of human perception and cognition.” 
However, Sack notes that information visualization methods, especially artworks 
that utilize the language of information visualization, in fact have the ability to 
allow viewers to observe things more accurately. That is, Sack shows that visu- 
alization can be used to see normal, expected things as unexpected; to increase 
our attention and focus, rather than dilute it. Specifically, he looks at projects 
that engage with infinite possibilities, that highlight the uncanny in normal inter- 
actions and behaviors, and that show the complexity and ideological structure of 
everyday activities [182], Although focused on art of of a political nature, Rita 
Raley, notes something similar: that art be uscful as a tactic to generate “the 
temporary creation of a situation” which allows users to observe “dominant nar- 
ratives” clearly, and that “engage in a micropolitics of disruption, intervention, 
and education” [173]. It is hard not to see a parallcl with Tilomas Kuhn, who 
writes that “even when the apparatus exists, novelty ordinarily emerges only for 
the man who, knowing with precisión what he should expect, is able to recognize 
that something has gone wrong.” That is, Kuhn summarizes, expectations obscure 
the ability of a scientist to make new discoveries [120]. And media projects that 
engage in visualizing and unveiling may be effective tools that could be applied 
to making more accurate observations about the world. 
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2.4.2 Hypothesizing 

Lev Manovich and others at the Software Studies Initiative at UC San Diego have 
written a series of articles and proposals describing thcir cultural analytics research 
program. Cultural analytics extends digital humanities analyses of texts by look- 
ing as well at visual new media artifacts such as video games, video clips, cinema, 
animation, and art, as well as by examining and analyzing data from human in- 
teraction with websites, software, and social media. Through using various data 
mining algorithms and irnage processing techniques, cultural analytics researchers 
transmute raw data into quantifiable representations of cultural phenomena. This 
increase in scope and interpretive techniques leads to useful visualizations of, for 
instance, of global flows of cultural change. It airns to create quantitative mea- 
sures of cultural innovation. These measures can be used to create visual maps, for 
instance, of “global cultural production and consumption” which are sufficiently 
temporally discrete so that a representation of the flow of change in various cul¬ 
tural médiums- rnusic, design, art, finalice- could be presented. The goal of the 
research program, as defined in the white paper, “Cultural Analytics: Analytics 
and Visualization of Large Cultural Data Sets” [144] is to create detailed Ínter- 
active visualizations of various cultural flows which provide “rich information” 
that can be presented in different formáis. These cultural data sets are conceived 
of in the same scope as other global data sets such as those created by various 
scientific endeavors. For instance, Manovich compares forrns of cultural data to 
real-time maps of global Computer networks and also to real-time maps of the 
range and intensity of an earthquake. Manovich makes a point of differentiating 
these dynamic contemporary data sets with the historical data sets that are more 
often used in cultural analysis within the humanities. The interdisciplinary nature 
of this nascent field draws on digital humanities, social Sciences, statistics, data 
mining, information visualization, and art. 

A primary question is whether or not this project is actually possible- Can 
cultural data be meaningfully extrapolated at all? Manovich discusses the (mainly 
technical) reasons why current data projects fail to provide effective global cultural 
analyses and points to possible ways to provide these. Looking at current art or 
information visualization projects, Manovich notes that they use relatively small 
amounts of data (compared to what is available via Googlc, Amazon, or what is 
captured from scientific sensors.) That is, they are constrained by the data, or 
at least the form of the data, rather than motivated by the “more challenging” 
theoretical questions and agendas that the creator might desire. Moreover, these 
projects generally do not engage in any sophisticated data analysis and also do 
not effectively layer múltiple sets of dynamic data. In particular, he points to the 
field of digital humanities and notes that the representations of textual data are 
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not transformed into “compelling visualizations,” and also that, inherently, the 
data sets are limited to texts (as opposed to images, statistical information, etc) 
and generally static, historical, canonical, “high-culture” texts at that. 

Manovich summarizes the “new paradigm” of cultural analytics by outlining 
a series of steps that encapsulate its agenda, which inelude: a focus on visual 
data; using extremely large contemporary global data sets; the use of statistical 
analysis, feature-extraction, clustering; formatting output to work on very large, 
high-resolution displays; and a focus on non-corporate agendas. As a overarching 
goal, he wants to be able to track and visualize the flow of cultural ideas and 
influences to provide “the first ever data-driven detailed map of how cultural glob- 
alization actually works.” A variety of other projeets are featured on the cultural 
analytics website, such as a visualization of comments on a set of interconnected 
MySpace pages, visual analyses of cartoons, films, and music videos, and an exarn- 
ination of art history through the computational analysis of 20 million paintings. 

The articles and white papers introducing cultural analytics tend to describe 
the research program in terrns of “explorations” of data, presentations of an 
“overview” of information, finding cultural “trends” and “patterns.” In other 
words, their work is in many ways analogous to the mathematician John Tukey’s 
work in exploratory data analysis [211] and digital humanities researcher Franco 
Moretti’s “distant reading” [153]. However, more recent work by So Yamaoka, et 
al., moves beyond the idea of simply gathering and visualizing cultural data sets. 
Instead, he discusses ways in which interactive visualizations could enable users to 
progressivcly develop and refine hypotheses to apply in gaining new insights into 
large collections of data [238]. Specifically, Yamaoka looks at analytic techniques 
that enable users to “gain new insights” into large databases of digital images, 
including “scatterplot múltiples” that allow users to simultaneously view different 
dimensions of the dataset in order to facilítate reasoning about the meaning of 
the data. 

2.4.3 Experimenting 

In an article discussing what he calis the “transpareney” experience, philosopher 
Alva Noé writes about the difficulties involved in “any attempt to make perceptual 
experience itself the object of investigation.” For Noé, an contemporary goal of 
art is to encourage us to “catch ourselves in the act of perceiving” [159]. Edward 
Shanken similarly examines the early association of media arts and cybernetics, 
explaining that a main concern of media artists is “to investígate the cognitive 
functioning of their own minds with respect to the processing of information and 
the creation of meaning” [191]. The projeets described earlier in this chapter are 
explicitly positioned as installations that opérate as experiments, collecting data 
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for later analysis. As I discuss in Chapter 5, an intriguing idea is to explicitly make 
use of a media arts project as a way to conduct rigorous, repeatable experiments 
in order to test specific hypotheses. 

Of course many aspects of scientific methodology are not normally carried out 
in an arts context; research in the arts is not tantamonnt to scientific research, 
yet artistic methodologies can have a direct contribntion to scientific methods. 
Clarifying what these artistic methodologies are and noting the points of contact 
with scientific methodology may be usefnl tool for mediation between participants 
with different research agendas within art-science projects. 

2.5 Conclusión 

In this chapter, I looked at a series of projects with clearly-defined artistic outpnts. 
I showed that even though these projects were not intended to have any research 
outputs, they nonethcless incorporated cnrrent research topics in a variety of do- 
mains, including Computer visión, information retrieval, and digital humanities 
computing. In particular, I reframed salient aspects of these projects in terms 
of the methodological thernes defined in the previous chapter. Specifically, I 
looked at: the ways in which Cell Tango supports generation and mediation tasks; 
how Data Flow generated novel research into an animated layont algorithm; and 
how the series of smaller pieces, Information Poerns, supported provocation tasks. 
Moreover, I explored how media arts project withont an interdisciplinary research 
agenda can support art-science projects. I showed that, contrary to comrnon 
thinking, media arts methods, even those geared toward an artistic ontpnt, share 
aspects with scientific methods, or at least could be seen as supportive of these 
methods. Specifically, I looked at ways in which media arts projects could enablc 
the tasks of observing, hypothesizing, and experimcnting. In the next chapter, I 
look at a project that was conceived as having both artistic and research outputs, 
and I discuss the rolc-switching that media artists perform in singly-anthored and 
collaborative projects to generate new ideas. 
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Chapter 3 

Fluid Research: Annular 
Genealogy and the Fluid 
Autómata Project 

3.1 OverView of Chapter 

In the previous chapter I investigated some ways in which media arts projects 
may have overlapping concerns with research topics in other fields. Moreover, I 
contended that aspects of media arts projects have methodological components 
that could contribute to research methods in these other fields, and to empirical 
methodology in general. In this chapter I look more closely at an interactive 
generative artwork, Fluid Autómata, that was initially formulated with hybrid in- 
tentions as a research project to create a fluid dynamics algorithm with particular 
properties and as an interactive media arts project that encouraged collabora- 
tive participation. I also discuss a subsequently devcloped audio-visual art piece 
called Annular Genealogy, based in part on the technologies invented for the Fluid 
Autómata project. Although these works have been presented in art contexts, I 
explore them here as examples that utilize the methodologies outlined in Chapter 
1. In particular, Fluid Autómata is presented as an example of generation. That 
is, although the project liad no specific research agenda to begin with, through the 
act of exploring aesthetic possibilities interesting areas of research were exposed. 
Moreover, through the artistic investigation, the contours of the boundaries of 
what was possible are defined. This chapter explores the process of identifying 
boundaries in order to define or hclp focus interesting or useful research ques- 
tions. In the case of Fluid Autómata, the research questions are in the areas 
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of software engineering, visualization, interaction, and fluid simulation. For the 
Annular Genealogy installation, the research is related to collaboration of a syn- 
chronized process using feedback from múltiple modalities of representation and 
interaction. 


3.2 Introduction to the Fluid Autómata Project 

The Fluid Autómata System explores the use of fluid simulation as an aesthetic 
component for interactive art. Whilc fluid simulations are often used in scientific 
visualization applications in order to give researchers a practical understanding 
of a fluid system within a particular context, the use of fluid simulations to cré¬ 
ate realism and excitement in entertainment contexts is also a widcly explored 
topic. Moreover, fluid Systems have been used as a technique in generative me¬ 
dia arts contexts, and a number of recent art projects utilizo fluid simulation as 
a component of the work. A method created by Computer graphics researcher 
Jos Stam in 1999 (and presented to the game developers community in 2003) to 
create a stable fluid system first rnade it possiblc to represent realistic looking 
fluids at real-time frarne rates [200, 201]. Many interactive artworks have made 
use of this particular technique. For instance, the Creative coder Memo Atken has 
created a series of demonstrations based upon Stam’s method, showcasing them 
using mobile devices for interaction and making the code available for OpenFrame- 
works and Processing multimedia frameworks 1 . Another project that incorporates 
Stam’s method is artists Graham Wakefield and Haru Ji’s Artficial Nature. This 
project uses Computer visión techniques to allow participants to interact with a 
3D fluid representation through the movement of their bodies [222], Other fluid 
simulation methods, such as [94], are optimized for real-time interaction in video 
games. Although simulation methods generally focus on producing accurate rep- 
resentations of natural Systems, the Fluid Autómata system demonstrates that 
aesthetically-interesting visuals with a wide variation of movement and color can 
be produced from a simple set of rules that do not attempt to exactly reproduce 
natural systems. In this sense, although influenced by physical simulation, Fluid 
Autómata could be considered a “generative art” system [81, 15]. 

Scientific visualization projects aim to hclp researchers identify and reason 
about salient aspects of their data. Whilc an aesthetic sensibility may contribute 
to the success of a visualization technique, this is not normally the primary moti- 
vation for its creation. Media artists, on the other hand, generally place aesthetic 
considerations at the forefront of their concerns. Similarly, physical simulations 
are concerned with accuracy and realism rather than extensibility and interaction, 

1 http: / / www .memo, t v/ofxmsafluid / 
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Figure 3.1: Examples of output from an iPad application utilizing the Fluid Au¬ 
tómata system showing the wide range of aesthetic possibilities generated by the 
system when using different fluid profiles defined by customiziug the fluid, visual- 
ization, and noise parameters. 


which are central to media arts. However, media arts projects, due to time con- 
straints or limitations in technical knowledge, often incorpórate readily-available 
techniques not originally intended for artistic production, and thus that are not 
necessarily easily adaptable to artistic situations. A goal of Fluid Autómata was 
to provide a system that provided more flexibility and that could be usecl in many 
situations, including on less powerful mobile devices, while retaining enough real- 
ism to be useful for simulations. Figure 3.1 shows examples of different outputs 
created with the Fluid Autómata system. 



Figure 3.2: Details of high-resolution output demonstrating “unrealistic” fluid 
simulation. The left image shows the addition of a high amount of vorticity; the 
right image shows the “spikiness” associated with a high amount of energy. 
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3.3 Technical Details of Fluid Autómata 

The Fluid Autómata System is composed of three integrated components: a fluid 
simulation engine; a flow visualization technique; and an interaction component 
that encourages interaction with the visnalized fluid simulation [74], Here I look 
at the technical details of the fluid simulation and flow visualization components 
that are shared across múltiple deployments of the project. 

3.3.1 Fluid Simulation 

The fluid simulation is a novel technique inspired by cellular autómata Systems, 
in which discrete cclls of information are updated by surrounding cells. As op- 
posed to early cellular autómata Systems in which each ccll contained only binary 
information, the Fluid Autómata systern contains 2 8 * 2 8 states, where there are 
256 valúes for the magnitude of a fluid vector and 256 states used to describe 
its direction. While the ñame of the systern was inspired by the biologist Tibor 
Gánti’s discussions of “chemotons” [83, 84], the initial kernel of insight for the 
Fluid Autómata systern aróse while thinking about how to create a simple rulc- 
based systern to produce emergent behavior. Cellular autómata Systems, made 
popular through the introduction of John Conway’s “Game of Life” [85], demón¬ 
strate complex behavior emerging through an iterative systern that updates the 
state of each elcment (positioned in a uniform grid) based on a set of basic rules. 
These rules determine the next state of each elcment by qnerying each of its 
neighbors. In Conway’s original autómata systern, each pixcl has a binary state, 
and either “lives” (is set to 1) or “dies” (is set to 0) based lipón the number of 
surrounding pixels that are on or off. In many implcmentations of the Game of 
Life, nsers interact with the systern by using a rnouse to tnrn pixels on or off. 
Other cellular autómata Systems, including many that are explored in Stephen 
Wolfram’s “A New Kind of Science” [233], explore different rule sets and involve 
múltiple states. 

The current versión of the Fluid Autómata systern effectively simulates the 
movement of fluid using an 8-bit cellular autómata systern that stores 256 states 
for orientation and 256 states for magnitude for each pixel. Retaining the dis- 
cretization and simplicity central to cellular autómata Systems allows us to create 
a complex systern that is linear and replicablc. Unlike the majority of connnonly- 
implcmented fluid simulations which utilize the non-linear Navier-Stokes eqna- 
tions, onr algorithm is always stable at any length of timestep. That is, onr 
systern is inhcrently non-realistic dne to the fact that there is no mass conser¬ 
varon condition and because the fluid is compressible. Although our systern is 
not physically accurate, it has the advantage of being easy to rnodify in real-time, 
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Figure 3.3: Example output using a custom color palette with high-contrast irnage 
Processing variables. 


and, more importantly, it is easy to comprehend and relatively straightforward to 
implcment, lcading to its incorporation in a range of projects. 

Since one of the goals in the creation of the Fluid Autómata systern is to 
emphasize creativity and interactivity, it ineludes a robust simulation engine that 
allows a wide range of fluid-like behaviors to be explored. For instance, the systern 
allows users to set parameters describing viscosity, rotational energy, and various 
momentum parameters. Various versions of this systern llave been implemented 
in the different projects that use the Fluid Autómata systern, taking advantage 
of available hardware on different devices. But, at its most basic, the systern 
distributes a flow of energy throughout the systern as follows: 

1. The screen is divided into a grid of cells. 

2. New energy is added into the grid by user interaction. 

3. The energy at each ccll is split into three streams, a forward strearn and a 
left and a right stream. 

4. In each of the defined directions for each stream defined in step 3, the energy 
of each ccll is moved into the neighboring cell via the following process: 

(a) The ccll is displaced along the vector describing the energy stream. 
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Figure 3.4: Photo of iPad application using the Fluid Autómata System witli a 
live video feed replacing the backgrouud noise texture. 
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original energy ratios split energy 


forward:side = 50%/50% 
left:right = 50%/50% 
angularity = pi/2 


forward:side = 70%/30% 
left:right = 50%/50% 
angularity = p¡/6 


forward:side = 20%/80% 
left:right = 15%/85% 
angularity = 3/4pi 




Figure 3.5: Examples of fluid Systems with different characteristics, i.e., fluid 
profiles, based on different settings. The settings can be changed in real-time. 
The left side of the chart represents the energy in a single ccll; the right side 
of the chart shows how the energy is split into different streams based npon the 
parameters defining the ratio between forward and orthogonal momentnm; the 
ratio between left and right momentnm; and the angularity of the orthogonal 
momentum. In the top row (a), energy with a magnitnde of 255 and an orientation 
of pi/2 is split evenly between forward and orthogonal momentum. In the niiddie 
row (b), most of the energy in the ccll remains moving in the original direction; the 
orthogonal energy is cióse to the forward orientation. In the bottom row (c), most 
of the energy is moving to the side, with a high angularity, and with an uneven 
distribntion between the left and right sides. (Note, the arrows representing energy 
vectors are not drawn exactly to scale.) 
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(b) For each neighboring ccll the displaced cell intersects with, create a 
“partial” vector by scaling the original vector with the amount of inter- 
section. 

(c) This partial is added to the ccll it intersects with. 

5. When this process is finished for the entire grid, the partials associated with 
each cell are combined, creating a new vector replaces the current vector in 
the cell. 

6. Energy is removed frorn the System by scaling the energy in each cell by a 
dampening factor. 

7. Steps 2 through 6 are iterated at each timestep until there is no energy lcft 
in the system. 

More formally, the fluid system acts on a grid G of cells C tJ each containing an 
energy vector E %y with a particular magnitude m and orientation 6, and where 
0 < i < columns and 0 < j < rows. The resolution of the grid depends on the 
effectiveness of the hardware. On a third generation iPad, the máximum resolution 
at real-time frarne rates is a 25x25 grid of cells; on a desktop Computer with a 
modern graphics card, a 100x100 grid of cells or greater will run at interactive 
frame rates. 

In order to define a fluid profile for the system, two ratios are introduced that 
regúlate that behavior of energy as it moves through the cells. The first, the 
momentum ratio, or ri, defines how much energy moves forward versus moving 
to the sides. The second, the directionality ratio, or r 2 , defines how much energy 
moves to the left versus moving to the right. A parameter, angularity. is defined 
as Z0 G [0,7r), describing a rotation offset from ¿.Q. Additional parameters influ- 
encing the fluid profile are the viscosity of the system, which acts as a dampening 
factor defining the rate at which energy is removed from the system, and the sensi- 
tivity which Controls how much energy is added to the system through sorne form 
of user interaction. New energy is added into the cells in a particular direction 
using the multitouch capabilities of the tablet device. The amount of energy that 
is added in a particular touch depends upon the sensitivity parameter, which can 
be adjusted during runtime. The magnitude of energy is also determined by how 
far the current position of the touch is from its previous position. This difference 
also determines the direction of the added energy. If there is no change in position, 
then the energy is added in the last known direction. This vector of new energy 
is added with any existing energy at the currently touched cell to update E i y 
At each timestep t during the operation of the fluid system, the energy E in 
Cíj is split into three sepárate streams: a forward stream, F, and two “orthogonal” 
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a b 




Figure 3.6: Example showing how a single stream of energy in a single ccll is 
distributed to other cells. In this example, the left grid (a) shows that the energy 
in cell 0 is pointing in the direction pi/4 with a magnitnde of 0.5. The dotted 
box shows where the “displaced” ccll intersects with its neighbors. The largest 
intersection of energy stays within the original cell (cell 0); a tiny amonnt of energy 
is pnshed into ccll 3; and a small amonnt of energy is pnshed into cells 1 and 2. 
The right grid (b) shows the distribntion of energy frorn this single cell into its 
neighbor cells and back into itself. 


streams, L and R. Using the cnrrent fluid profile (the valúes for the parameters 
of momentum, directionality, and angulañty ), and the cnrrent magnitnde and 
orientation for each ccll, I define these three streams like so (in Eqnations 3.1 
throngh 3.3): 


F 

L 

R 


r i rn \ 

. 9 J 

(3.1) 

(1 - r 2 )( 1 - ri)m\ 
d + 0 ) 

(3.2) 

r 2 ( 1 — ri)m\ 

0-$ ) 

(3.3) 


Figure 3.5 shows examples of how the cnrrent total energy in a cell is split into 
three streams based on these parameters. Every ccll in G thus contains three 
sepárate streams of energy, F, L , and R. These streams are used to define the 
flux of energy moving from each ccll into its neighboring cells. 

A “displaced” cell D(C, v) is a copy of a ccll C e G that moves along a vector 
v positioned at the center of C. This displaced cell D intersects with between 
1 and 4 cells (the original ccll C itself and up to three neighboring cells). The 
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Figure 3.7: Users gathered around a multi-touch table running a project that uses 
the Fluid Autómata System. 


magnitude of any energy vector is constrained to range between 0 and 1. When 
displacing a ccll, the magnitude is scaled by the length of a ccll side. For instance, 
if the grid is divided into 10x10 cells, then each cell side is .1 in length. And so 
the magnitude in a ccll is scaled by l/10th before being displaced. Tliis ensures 
that a displaced ccll will only ever intersect a ccll that is its immediate neighbor 
(although exceptions to this constraint may have interesting Creative possibilities.) 
The amount of overlap between the displaced ccll and the cell it intersects with 
determines how energy is placed into that ccll. An intersection Í(D,C) is simply 
the amount of overlap between the displaced cell and another ccll (Equation 4). 
The valué of any intersection can range between 0.0 (no intersection) and 1.0 (full 
overlap). 

I(D,C) = ^Area(D) D Area(C)^j / Area(C) (3.4) 

This valué is used to create a “partial” vector p via a function P(N, v, C ) for each 
energy stream. This partial is calculated simply by scaling the stream by the 
amount it overlaps with the current ccll, as defined in Equation 5: 

P(N, v, C) = I(D(N, v), C ) * v (3.5) 

In Equation 5, N refers to a neighbor cell of a ccll C G G and v refers to one of the 
energy vectors (F, L, or R ) in that neighbor ccll. Figure 3.6 depicts an example 
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of this displacement and the subsequent generation of partíais. For each cell 
in G, each neighbor is examined to determine how mnch each of its three streams 
overlap. The partíais created by any intersection between the neighbor streams 
are snnnned to generate the new energy vector for the cell (Eqnations 6 throngh 
9). It is important to note that a cell is considered to be a neighbor of itself. For 
instance, if a vector of magnitude 0.5 is pnshed upwards at 90 degrees, it would 
intersect with both the cnrrent cell and the neighbor cell above it. Since the 
displaced cell would rnove a distance of 50% of its hcight from its current position, 
it would end up intersecting the current cell and the neighbor cell equally, and thus 
a copy of the vector, scaled by 50%, would be placed in each of the cells. Figure 
3.6 shows a typical example of a single stream of energy in a single cell being 


distributed to its neighbors. 

Excluding any dampening factor or 

any new energy 

added from user interaction, the system will retain exactly the 

same amount of 

energy over each timestep. 

i+ 1 3+ 1 


Fi,j,t = 

5] X P(C„, Ctj) 

(3.6) 


P=i- 1 1=3- 1 



i+ 1 3+ 1 


Fi,j,t = 

P{Cp,qi Cij) 

(3.7) 


P=i~ 1 1=3~ 1 



i+ 1 3+1 



'y ^ y ^ P{Cp,qi Rp,q,t-liCi,j) 

(3.8) 


P=i- 1 1=3-1 



Pi,j,t T R%,j,t T Ri,j,t 

(3.9) 


Other parameters can also be adjusted to create different characteristics for a 
particular fluid profile. These inelude: controlling the “jitter”, or randomness of 
the system, and clamping the máximum outflow for the cells within the grid. 1 
also experimented with a toroidal representation of the system where fluid energy 
wraps around the edges of the screen, instead of bouncing off the edges. Setting 
a máximum outflow parameter interestingly creates the sense of ice cracking and 
melting when a particular threshold is exceeded. And different settings of viscos- 
ity and angularity can create more or less turbulent behaviors. Whilc it may be 
surprising that a simple heuristic could mimic the complexity of fluids, the itera- 
tive nature of the system does in fact create a wide variety of fluid-like structures, 
including the creation of eddies, vórtices, and turbulence. Figure 3.1 and Figure 
3.2 show examples of fluid systems with different fluid profiles defined by different 
settings for the angularity and directionality parameters. 

Just as simulations for realistic films and video games do not feel constrained 
by a perfect representation of the physics of a visual effect, so to should media 
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Figure 3.8: Schematic for the main functionality of the Fluid Autómata System. 
The output texture after one timestep becomes the input for the next timestep. 


artists not feel constrained by a perfect representation of existing algorithms and 
equations for a particular kind of effect. By creating a custom fluid System with 
a wide range of parameter adjustments I was ablc to extend the aesthetic applica- 
bility and variation of the fluid System to capture unusual behaviors not normally 
depicted with fluid representations. Although the System appears realistic, it in 
fact sacrifices physical accuracy in order to emphasize interactivity, expressivity, 
and experimentation. 

The following code listing, UpdateCellVector, describes the fluid sirnula- 
tion, based on CA, as described in section 3.1. The cell parameter refers to a cell 
positioned in the CA grid (pos), with a particular state described by a vector ( vec) 
that has a particular magnitude ( mag) and orientation (ang). The grid, has a pixcl 
width of width and pixcl height of height, and a particular number of colunms 
(cois) and rows (rows), and contains a double array of cclls. The variable m refers 
to the ratio that splits a cell vector into forward and “orthogonal” streams, the 
momentum parameter, or ri. The variable (j) refers to the angularity parameter. 
Other optional variables, such as the directionality and viscosity parameters, have 
been excluded for clarity. Note that the NeighborCells procedure retrieving a 
ccll’s neighbors also ineludes the cell itself. Finally, the IntersectRects proce¬ 
dure (referred to but not shown) returns a valne between 0.0 and 1.0 indicating 
the amount that two rectangles of the same size overlap. 
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Figure 3.9: Photograph of viewers wearing 3D active stereo glasses within an 
installation of Annular Genealogy inside the AlloSphere Research Facility at UC 
Santa Barbara. 


Algorithm : UpdateCellYector (cell,grid,m,<j)) 


comment: Update vector vec in a Cell c by querying neighbors in Grid g 

procedure NeighborCells(c, g) 
neighborCells <— Array(9) 

for i i - 1 to 1 

for j i - 1 to 1 

' nc <— c.col + i 
nr c.row + j 
if nc <0, nc •$— grid.cols — 1 
do if nc > grid.cols — 1, nc 4— 0 
if nr <0, nc 4— grid.rows — 1 
if nr > grid.rows — 1, nr 4— 0 
^neighborCells,push(grid[nc] \nr\) 
return ( neighborCells ) 


procedure MakeRect(p) 
w 4— g.width/g.cols 
h 4— g.height/g.rows 
pLL vec2(p.x — w/2.0,p.y — h/ 2.0) 
pUR vec2(p.x + w/2.0 1 p.y + h¡ 2.0) 

return (Rect (pLL,pUR)) 

o9 

procedure Partial(pos, nc, nu) 

p •(— IntersectRects(MakeRect(pos), MAKERECT(nc.pos + nv)) 
return (nv * p) 
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3.3.2 Flow Visualization 

A perennial concern of scientific visualization is the effective visualization of salient 
features of a vector field, as indicated by the wide variety of approaches to their 
representation [148]. A popular technique introduced in 1993, called Line Integral 
Convolution, effectively identifies detailed curvature features of a vector field. In 
this technique each pixel of a background image is filtered along “streamlines” de- 
fined by the vector field [22], Another early technique, Choreographed Image Flow, 
describes using image warping to generate animations for an animated representa¬ 
tion of flow-ficlds [196]. A more recent technique, Image Based Flow Visualization, 
represents flow using the iterative deformation of a texture mesh along the direc- 
tions of the vector fields. In this technique, an image is blcnded together with the 
distorted versión of itself at each frarne, causing pixels to “smear” and appear to 
leave streaks in the direction of the flow vectors [216]. While the creators of these 
techniques recognize and discuss applications outside of scientific visualization, 
more recent papers examine the relationship between aesthetics and visualization. 
For instance, [115] specifically looks at the various stylized qualities involved in 
painting and the possibility of brushstroke techniques for inspiring more effective 
scientific visualization methods. And [157] introduces a technique that allows 
artists to control small-scalc animations on “advected” textures. 

The main image processing scherne in the Fluid Autómata System is based on a 
feedback loop whereby a high-resolution background image is perpetually blcnded 
together with a distorted versión of itself. The characteristics of the distortion are 
based directly on the current state of the fluid System. This System is similar to 
van Wijk’s Image Based Flow Visualization, which has been extended for use in 
a variety of scientific visualization applications, including animated and 3D flows 
[207, 216]. Since the focus of the application is aesthetic exploration, I provide 
the user with a variety of tools to alter aspects of these blending operations, and 
in addition introduce an image processing layer whereby the user can change a 
variety of parameters, including: the rate and amount of blending, the type and 
quality of the background texture, and the brightness, contrast, and saturation of 
the blcnded image. The default background texture is a grayscale noise texture 
at a resolution exactly matching the display size. However, I have experimented 
with various types of background textures, including lower resolution textures, 
static colored textures, static image textures, dynamic textures that are updated 
by noise functions, and dynamic textures that are updated by a live video feed. 
Figure 3.3 shows an image created using a static colored background texture 
with high contrast and high saturation image processing parameters. Figure 3.11 
shows an image with no saturation and that uses a low resolution, black and 
white background texture to create an interesting “smearing” effect. And Figure 
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3.4 shows an image that is created using a live video feed as the background image, 
rather than an image populated with randomly-colored pixels. 

Figure 3.8 provides an overview of the fluid visualization process over the 
course of a single frame. At the system: a) distorts the previous image texture 
(if i > 0, otherwise it uses a copy of the background texture) based on user 
interaction and the current fluid profile; b) blcnds the background texture in 
with the distorted texture based on a blending parameter (that can be updated 
in real-time by a user); and then c) applies image processing filters (based on 
the image processing parameters described above) to the image to create a final 
output texture for this frame at t i+ This output texture is then used as the 
input texture for the next frame. 

3.4 Deployment 

3.4.1 Fluid Autómata, iOS Application 

An initial iteration of the Fluid Autómata system was implemented in a stand- 
alone versión of Fluid Autómata that could be run by a single user on an iPhone 
or iPad. Although other fluid simulation applications already existed for the iPad, 
ours offered a wide variety of customizability, including the ability to change 
image processing parameters and fluid profiles. Additionally, users were able to 
choose presets to quickly jump to interesting states and motions. Finally, the 
application included the ability for users to turn on the video camera to capture 
a live feed of themselves or their surroundings and to superimpose that into the 
fluid system. The application received positive reviews on blog sites, including 
CreativeApplications.net and OneAppOneDay.com. Within a few months it liad 
a five-star rating and was downloaded thousands of times. 

3.4.1.1 Interaction in the iOS application 

The primary interaction component utilizes the multitouch capabilities of iOS 
devices, creating a believable representation of fluids being disrupted by human 
touch. Additionally, users are able to manipúlate a wide number of parameters to 
control the fluid profile of the simulation as well as a number of image processing 
components. Much experimentation went into making the user interaction with 
the fluid system feel responsive and inviting: by tapping the screen the user adds 
energy to the system; moving a finger across the screen overrides the fluid dynamic 
system by forcing the vector to move in the indicated direction; and múltiple 
fingers can be used to push energy around in a more complex way, possibly by more 
than one person. Other gestures can also be enabled to update fluid properties 
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or image processing parameters. For instance, a pinching gesture using all five 
fingers simultaneously causes the entire background texture to scale up or scale 
down, creating a zooming effect. Similarly, a five-fingered panning gesture causes 
the entire background texture to be translated in the direction of the pan (as 
deter min ed by the centroid of the five fingers), shifting all of the fluid vectors so 
that they point in that direction. In addition to being able to add energy to the 
System, users can update the fluid parameters and image processing parameters 
in real-time, or select a specified fluid profile defined at a previous time. In the 
iPad versión of the project users can doublc-tap the screen to bring up a set of 
controllers that affect the various parameters of the system. Figures 3.10a and 
3.10b show a detail of sliders affecting the image processing parameters and fluid 
parameters, respectively. Users can also save the current fluid profile at any time, 
allowing it to be quickly retrieved in future sessions. 
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Figure 3.10: Details of the iPad controller. The sliders are used to update the 
image processing parameters (left) and the fluid profile parameters (right) in real- 
time. 


3.4.2 Fluid Autómata Performer, iOS Application 

The current iteration of Fluid Autómata is a more full-featured versión that is 
intended for performance situations, for example, by choreographers or VJs to 
augment a dance performance or musical event. In this versión, the iPad is con- 
nected to a projector or monitor that displays the fluid simulation. The actual 
iPad screen then becomes a rich interface for controlling many aspects of the 
system. In addition to allowing for the control of fluid vectors, the user can in¬ 
corpórate video playback, a live video stream, or dynamic noise textures. The 
system also can be set up to receive or send OSC messages corresponding to dif- 
ferent aspects of the Fluid Autómata system so that it can be used as a more 
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Figure 3.11: Example output of visualization using low-resolution binary back- 
ground that creates a sharp, smearing effect. 


general controller for other artistic outputs, or to be controlled by other external 
devices on the same network. This versión also enables the ability to do real-time 
capture of a Fluid Autómata session, which can then be shared or modified in 
video editing software. This versión of Fluid Autómata is available for the iPad 
and iPhone via the iTunes App Store 2 . 

3.4.3 Collaborative Installation 

An iteration of the Fluid Autómata project was showcased at the 2011 IEEE 
VisWeek Art Show, curated by Bruce Campbell and Daniel Keefe [69]. Research 
investigating interactive flow visualization indicates that novel interaction tech- 
niques can successfully enable collaboration and enconrage exploration [103]. In 
order to investígate these aspeets of fluid simulation, this versión was presented 
as an interactive installation for múltiple users. Each user was given an iPad and 
invited to interact with a single fluid systern projected large-scale on a wall. On 
each iPad, users saw the current state of the fluid system’s underlying vector rep- 
resentation (represented by rotating and scaling triangles) as well as an indication 
of the other user’s interaction (represented by colored circles). This was done in 
order to see how users would interact together to manipúlate the fluid systern. 

2 http: //www. fluidautomata.com 
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An interesting outcome of this presentation was the realization that many peoplc 
were very interested in how others were interacting, and that allowing people to 
see each other’s interactions increased engagement with the installation [73]. 

3.4.4 Multitouch Table 

A more recent iteration of the Finid Autómata system was ported to a multi¬ 
touch table and shown in an art show at the University of Arizona called ”Form, 
Content, and Computation.” This installation made it easier for múltiple people 
to interact with the system at the same time, and to explore specific gestures that 
were defined to allow users to alter the intensity of the turbulence of the system. 
In this installation, I used the TUIO protocol to map touches on the table as 
inputs that updated the Fluid Autómata fluid vectors. 3 The table was built by 
the engineer Matt DePorter and uses a perimeter of LED light sources and light 
detection sensors to detect touches on the surface of the table. I created a timcline 
that cycled through different fluid profiles so that users could experience different 
aspects of the system without having to worry about any other interactions except 
multitouch to move fluids arouncl. Figure 3.7 shows an irnage of people interacting 
with the Multitouch Table running an iteration of the Fluid Autómata project. 

3.4.5 Virtual Space Installation 

Another iteration of the Fluid Autómata system utilizes the gyroscope sensor built- 
in to the iPad in order to transform the tablet’s screen into a “magic lens.” In this 
versión, the fluid system is placed on the six sides of a cube map (also known as a 
“skybox”), a technique often used in video garnes to display landscapes or distant 
horizons. The user is imagined to be inside a virtual fluid system that they can 
see by spinning in place while rotating the tablet. By touching the screen, a ray 
is sent to the cubemap texture, allowing the user to interact with the surrounding 
fluid system. Figure 3.12 shows a series of photos taken of a user interacting with 
the virtual fluid system. 

3.4.6 Study in Brownian (F*) Motion 

The Fluid Autómata system has also been adapted for use as an instrument for 
controlling audio-visual compositions. In this configuration the output of the 
application is projected onto a large display. In addition to being controlled by 
multitouch, the system can respond to Open Sound Control (OSC) messages sent 
by another Computer that are generated by musical events. Additionally, fluid 

3 http://www. tuio.org 
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Figure 3.12: A user interacting witli the virtual Fluid Autómata installation using 
the iPad as a magic lens. 


vectors and various fluid parameters can be transmitted wirelessly via OSC to 
influence the composition. I have also experimented with attaching piezo sensors 
to the iPad itsclf in order to directly input data into an algorithmic composition 
engine. The first iteration of the Fluid Autómata System was a collaboration 
with the composer, Kiyomitsu Odai, in the Spring of 2011 at the media arts 
showcase, “Questionable Utility,” at University of California, Santa Barbara. In 
this improvised performance, Kiyomitsu Odai was responsible for creating live 
audio based on a generative System, while I used the multitouch capabilities of 
the iPad to push fluids around in response to the audio. Figure 3.13 show a photo 
of the performers interacting with the composition. 

3.4.7 Annular Genealogy 

Kiyomitsu Odai and I also collaborated on a more ambitious multimedia project 
involving generative music. We talked at length about the differences between 
this iteration and the previous, and decided that we wanted to create a piece 
where the sonification and the visualization were different representations of the 
same system, rather than two sepárate systems presented together. That is, we 
wanted to explore what would happen when the the data used to update the 
visualization system was the same data that updated the composition system. 
I defined a networked protocol based on OSC in order to transfer information 
back-and-forth between the different computers (one running Fluid Autómata 
system, and the other the musical composition). Rather than having a centralized 
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Figure 3.13: Photo from a live performance of Study in Broumian (F*) Motion. 


platform for generating the data, the data was generated in a constant multi-modal 
feedback loop between the two computers. After being seeded with either an 
initial sounds or an initial fluid vector, the System could run on its own indefinitely. 
Additionally, we could use interaction to interfere with the endless chaotic looping 
of the system. We narned the piece Annular Genealogy , highlighting the evolving 
“families” of feedback loops (annular means “ring-like” or “ring-shaped”). Sorne 
other additional components of Annular Genealogy that differentiated it from our 
previous work, Study in Brownian (F*) Motion , included the use of 3D space, the 
use of dynamic background textures based on Perlin noise functions, and the use 
of microphones to pick up ambient noise which were fed into the fluid system [76]. 
The piece has been presented as a live performance (it was performed it at the Bits 
& Pieces media arts festival in 2012) and also as a stand-alone fixed audio-visual 
piece that was shown in the Allosphere Research Facility in 2012 [2], Figure 3.9 
shows audience members inside the immcrsive Annular Genealogy installation. 

3.5 Fluid Interaction and Visualization Research 

In tliis section describe some of the research goals and design decisions involved 
in creating Fluid Autómata. A main motivation for the project was the desire to 
better understand fluid dynamics, and in particular, to be able to create a custom 
fluid simulation that could be explored using mobile devices. Cellular autómata 
approaches to fluid simulation have been explored. An early paper by Gerard 
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Vichniac describes discretized models for fluid simulation based of Cellular Au¬ 
tómata, but indicates specific necessary parameters to avoid “divergences” [219]. 
Another paper surveys different cellular autómata paradigms for fluid simulation, 
comparing lattice gas, digital fluid, and lattice-Boltzmann strategies [25]. A 1998 
paper describes 3D flnids that can be represented with hexagonal and rhombodo- 
decahedron grids instead of uniform sqnares or cubes [16]. And Stephen Wolfram 
also explored general non-linear approximations of fluids via cellular autómata 
[232], More recent work examines cellular autómata for a wide range of tasks 
beyond that of simulation, including, for example, image processing [179], urban 
design [209], and cognitive Science [236]. 

Although a wide range of interactive cellular autómata Systems are availablc 
online, 1 was unable to find one that attempted to rnodcl fluid structures. After 1 
developed the fluid simulation component that ran at real-time rates, 1 began to 
explore a variety of different interactive techniques. These included the use of the 
gyroscope within the iPad device, the attachment of piezo microphones, and the 
use of multitouch for the iPad versión and the tonch tablc versión. While I did 
not investigate research into responsive or realistic touch interaction, 1 spent a lot 
of time testing a variety of parameters related to multitouch interaction with the 
fluid System, including how much energy a single touch added the System, how to 
control the direction of the energy, and how quickly the energy would be added 
to the system. While these explorations were not precisely measured, 1 included 
interactive sliders that let users change some of these variables, and this could be 
used as a springboard for fntnre research into interaction. 

Another goal of Fluid Autómata was to make it easy to create different fluid 
profiles. In particular, I wanted the user to be able to add arbitrary amount of 
turbulence and vorticity into the fluid system. A recent paper describes a method 
to enhance fluid simulations with “intermittent turbulence” [241], Recent research 
has investigated how to add pleasing effects to fluid simulation for use in animation 
and film. A 2009 paper introduces a to define “Eulerian vortex sheets” that make 
it possible to stretch and wigglc fluids using a “liquid-biased filter” [113]. And [242] 
looks at preserving vortical detail in smoothed-particlc hydrodynamics techniques. 
By introducing the angularity, directionality, and momentum parameters, users 
could explore wide range of possibilities for creating turbulence and for introducing 
vorticity into the system. On the iPad versión, 1 added presets that allowed the 
user to immediately jurnp to fluid profiles that liad particular characteristics. For 
instance, the presets “Milky Way” and “Nebnlae” create roving eddies that appear 
and dissipate repeatedly. 

Finally, 1 wanted to make it easy to change the visual representaron of the 
fluids in order to highlight particular aspects of the fluid dynamics via changes 
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in, for example, color, saturation, and contrast. Research in the scientific visu- 
alization community has explored various techniques to visualize fluid flows. A 
survey of flow visualization techniques by Robert Laramee, et ah [123], ineludes 
dye-injection, texture advection, diflerent kinds of arrows, pathlines, streaklines, 
as well as the streamlines introduced by Brian Cabral and Leith Leedom in their 
paper, “Imaging Vector Ficlds Using Line Integral Convolution” [22], For this 
project, I decided to modify a texture advection technique callcd Image Based 
Flow Visualization, since it liad the benefit of being relatively fast since it re- 
quires only a shift in texture coordinates, rather than needing to sample a large 
number of individual particles [216]. By creating a pipcline of programmablc 
shaders, I made it easy to alter a wide range of parameters that lead to a wide 
range of visual effeets. 

As in the projeets described in Chapter 2, this chapter shows that a media arts 
installation inherently engages with current topics in research. Differently from 
those projeets, Fluid Autómata more intentionally created the first deployment 
as a way to conduct research into fluid dynamics Systems. This was partly as a 
learning experience for myself, as a way to explore existing techniques of model- 
ing fluids in real-time. But once I realized that most fluid dynamics algorithms 
used in media arts projeets were based off of the same algorithm and liad some 
limitations as far as Creative flexibility, it led me to conceive of my own poten- 
tial research contributions. Sonie of the many related topics I was interested in 
when the initial exploration was initiated included: collaborative interaction; mo- 
bile devices; musical interfaces; interactive graphics; and live performance. Whilc 
these may seem to form a disparate collection, they all relate to an interest in 
novel, easily-accessible forms of Creative interaction. The Fluid Autómata system 
functioned largely as a generator of múltiple, simultaneous, integrated research 
ideas related to these areas; the exploration of which fed back into the design of 
the Fluid Autómata system. Media arts projeets, though their goal is oriented 
toward the creation of a project, a performance, or a tool, can inspire novel re¬ 
search development in the areas the art piece draws from. This switch from an 
art production-goal to a research-goal results in a fundamentally diflerent orien- 
tation. Commonly in art production and design, a role of the creator is in part to 
figure out what is possiblc and to make do with the limitations and constraints 
of available resources and technology. A research agenda on the other hand in- 
volves exploring technologies with the goal of pushing beyond current limitations 
and constraints. This dichotomy between what I am calling a research-orientation 
versus a production-orientation is similar to the differences between Science and 
engineering. And in fact, this mediation between research and engineering is ex- 
actly what a media arts project can, idcally, enable. This role switching between 
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research and production is common in media arts projects, and can be explicitly 
directed to have outputs in both artistic and scientific realms. Indeed, this rolc- 
switching is powerful justification for the creation of media arts projects and their 
inclusión in art-science endeavors. 


3.6 Collaborative Multimodal Composing 

A secondary avenne of research that aróse through the Fluid Autómata project was 
about multimodal collaboration. As mentioned above, one of the more recent de- 
ployments of Fluid Autómata is its integration in the Annular Genealogy project, 
created in collaboration with Kiyomitsu Odai. In this section, I describe the in- 
triguing ramifications of thinking of the audio and visual components of the piece 
as different representations of a single multimodal source. Annular Genealogy is an 
interactive multimedia composition for two performers using multi-channel speak- 
ers, a projector, and a tablct Computer. The performance is organized around a 
generative music composition and its visual analog. Both the audio and visual 
components are explorations of feedback processes that encourage the perform¬ 
ers to interactively shape aleatory elements and transmute them into appcaling, 
transient structures. The composition engine works with a stochastic sequencer 
that uses Brownian motion as a guiding metaphor. Similarly, the visualization 
engine depicts colored fluid energy as a representation of dynamic, ephemeral 
structures. In addition to exploring these feedback processes independently of 
each other, each engine also directly influences the other via networked connnu- 
nication: both the visual and audio processes broadcast data via OSC messages 
which then influence various parameters of the composition and/or visualization. 
Finally, even the physical interactions are fed into the generative System as con- 
tact microphones are used to pick up the tapping and other ambient sounds made 
during the interaction. The ultimate goal of the performance is to bring various 
layers of feedback into a cohesive compositional experience. These feedback layers 
are interconnected, but can be broaclly categorized as physical feedback, internal 
or digital feedback, interconnected or networked feedback, and performative feed¬ 
back. Specifically, these layers inelude: the generation of new musical motifs 
being created from the processing of the output sound; the generation of visual 
forrns from the processing of the output graphics; the vector positions that govern 
the displacement of the visual forrns used as inputs to control music parameters; 
and the sequencing parameters controlling the generation of the composition used 
as inputs to control image processing parameters. 

In addition to having cybernetic properties of interlaced feedback Systems, the 
piece can be characterized as being fundamentally synaesthetic. That is, the mix- 
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Figure 3.14: High-level overview of the Annular Genealogy project showing pro- 
cesses at three tiers: performer, Computer, and audience mernber. The darker 
lines indícate the rnain feedback loops where the output of one process in piped 
in as the input to another. 


ing of the mutual generative processes conflates the aural with the visual modality 
and vice versa [45]. Throngh the continnons interlinking of the two engines (via 
the performers and via the data sent over the network) a single interconnected 
mnltimodal signal is created. The output of this signal is represented simultane- 
ously in múltiple domains. Figure 3.14 shows a high-level chart of the relationship 
between the performers, the audience, and the visualization and composition en¬ 
gines. The performers input information to software in parallcl using live coding 
and the multi-touch capabilities of the iPad. The outputs of each of the software 
engines then are fed back into themselves and into each other in various ways. 

By supplying a multi-touch and live coding environment as an interface to and 
influencer of the generative processes the project ineludes another layer of feedback 
in which the performer is ablc to respond to and shape the multimedia output. 
That is, the performers are conceived of as participants in a compositional process 
rather than as on-the-fly creators of audio-visual output. The generative software 
serves to create sorne structures independently of the performers; it is the role of 
the performers to guide the generational processes toward more compositionally 
interesting output and away from output that is overly repetitive, monochromatic, 
garish, or otherwise less satisfactory. Likewise, the audio and visual engines, via 
the various feedback processes, continually push against the explicit control of the 
performers. Overall, the composition is defined by a network of nested feedback 
loops that link the performer and the algorithm to create an inherent aesthetic ten- 
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sion between the generative and the interactive, the performed and the composed, 
the random and the intended. 



Figure 3.15: OverView of the interconnected components that comprise the cornpo- 
sition engine. The darker linc indicates the main feedback loop where the output 
audio signáis are recapture by microphones to be used in the generation of sound. 

Our composition refers directly or indirectly to a number of previous installa- 
tions. Compositionally, we were inspired by composer David Tudor’s Rainforest 
IV, an early sound-art installation featuring an entirely analog feedback System. 
Rainforest IV is “a collaborative environmental work, spatially mixing the live 
sounds of suspended sculptures and found objects, with their transformed reflec- 
tions in an audio System” [101]. In particular, Annular Genealogy extends the 
concept of using loudspeakers as “an instrument unto itself” (rather than a tool 
of replication) and of using feedback as a compositional source. Another overt 
influence on Annular Genealogy is 20th-century composer Iannis Xenakis’s con¬ 
cept of “Stochastic Music.” As described below, our stochastic sequencer is an 
integral part of the generative composition. In particular, Xenakis draws a paral- 
lcl between his compositional methodology and such natural phenomena as “the 
collision of hail or rain with hard surfaces, or the song of cicadas in a sumiller 
ficld” [237]. Similarly, we use a circular, stochastic timclinc that is elasticized by 
Brownian randomness to create foreign sounds that nonethcless have the feel of 
natural phenomena. Xenakis, in describing his landmark orchestral composition, 
Metastatis, hypothesizes that by constructing acoustic spaces of constant expan- 
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sion out of long passages of weaved glissandi “one can produce ruled surfaces by 
drawing the glissandi as straight lines” [237]. However Xenakis’ straightforward 
mapping of these glissandi sweeps to a chromatic scale is somewhat problematic 
as it imposes a non-linear relationship between the input and output frequencies. 
Our approach to this linearity problem, described below, involves replacing the 
chromatic scale with the Bark scale, which is at least psycho-acoustically linear, 
and thus more effectively captures the intended naturalistic feel. Other recent mul¬ 
timedia installations have also featured generative compositions that rnade use of 
feedback mechanisms between the audio and visual components. For instance, the 
New Zealand media artist Karen Curley created a work called Licht und Klang, 
an audio-visual installation that generates sounds via optical sensors that use the 
refractions of light through oil and water as inputs into sound generation software 
[44], Various clectro-acoustic ensembles have explored the use of networked feed¬ 
back as a tool for improvised performance. Most famously, The Hub creates mul¬ 
timedia performances based on sets of rules that transform signáis passed between 
performers and that are thcn presented in aural and visual domains [93]. A wide 
range of works have explicitly explored the notion of synaesthesia in installations. 
For instance, Jack Ox and David Britton’s 21st Century Virtual Reality Color 
Organ uses visual representations of sound waves as an element in creating an 
interactive landscape [162], More recently, Daniela Voto’s Multisensory Interac¬ 
tive Installation explores the interactive sonification of Kandinsky paintings [221], 
Other works invoke Michel Chion’s concept of synchresis to describe the “welding 
together” of auditory and visual phenomena [33]. For instance, Niall Moody’s au¬ 
diovisual instrument, Ashitika, generates simultaneous multimodal output from 
single gestures [152], Our work similarly creates synaesthetic output based on a 
synchretic fusión of a mixed audio and visual feedback loop. 

3.6.1 Compositional Details 

Annular Genealogy is made up of two distinct software engines, one governing 
the aural domain and the other the visual. The visual and aural components are 
related by the structural mechanisms of generating and processing feedback and 
by the thematic focus on generating organic structures that continually devolve 
and transform into new structures. Both the aural and the visual engines represent 
the movement of energy through a System. These software engines are completely 
decoupled, but influence each other via the output of different multimedia data 
transmitted via OSC messages. In this way, each of the engines becomes part of a 
component of each other’s feedback loop. In this section, I describe the individual 
iterative feedback processing for the composition and the visualization engines, 
and also indicate where output is sent to and received by the engines. 
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The generative composition is largely generated through the receiving and 
Processing of feedback from various sources: from the performer, frorn pararne- 
ters received from the visualization engine, from the audience interaction, and via 
the piping of the composition data back into the composition engine itself. The 
external inputs are directly provided by the output of the visualization engine, 
performer interaction via live coding of a SuperCollider script, and contact mi- 
crophone inputs that capture ambient sound. Moreover, the composition process 
is based on the continual recycling of audio data that is iteratively fed back into 
the microphones. Figure 3.15 shows a more detailed diagram outlining the main 
components of the composition engine. 

The composition engine is written entirely in SuperCollider 3 and consists 
of various interrelated components, including: the interactive timbre generator, 
the stochastic sequencer unit, and the Bark scaling unit. The interactive timbre 
generator Controls the overall quality of sounds by convolving the output of a 
compressor unit generator with a sine tone oscillator. In addition, a parametric 
equalizer and a ring modulator further ornament the signáis before and after the 
convolution. The stochastic sequencer unit Controls the timing and frequencies 
following a series of compositional heuristics. These frequency valúes are then 
piped into the Bark scaling unit, which defines the mapping of the frequency 
valúes according to the perceptual lincarity of human ears (described below). 

The main input signal is captured by a number of contact microphones, pass- 
ing through a 6-band parametric equalizer that manipulates and enhances the 
timbral variety of the signal. This signal is then densely compressed, generating 
a thick feedback loop: the compressor amplifies the softer signáis so that they 
are loud enough to be fed back to the loop, and at the same time, it squashes 
those above the set-up threshold in order to make the output sounds controllable. 
The feedback loop further functions as a distortion box by convolving sine tones 
with the output signal from the compressor. At the final stage of the System, the 
signal is ring-modulated with the low-frequency oscillator (LFO). The process is 
then iteratively repeated in real-time. The output through the speakers is again 
picked up by the contact microphones and becomes the main component of the in¬ 
put signal for the next pass of interactive timbre unit’s feedback loop. Interactive 
control of this feedback loop is available through the live coding environment and, 
additionally, simply by making sounds that will be captured by the microphones. 
In our original performance, for instance, contact microphones were attached to 
the iPad controller to use the percussive tapping of the visual performer as another 
input into the composition. 

The sequencer unit triggers the data for the following elements: the frequencies 
and durations for the enveloped sine tones, the six passing center frequencies and 
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their bandwidth as O valúes for the parametric equalization, LFO frequencies for 
the ring modulation, panning valúes for the spatialization, and the amplitude of 
the output sounds. All the valúes for those sequenced parameters are generated 
through stochastic processes that are based on four modes of increasing random- 
ness. In mode 0, a sequence of valúes is created through a simple rising motion 
(which is not random). In mode 1, Brownian motion is used, where each follow- 
ing number is either incremented or decremented only slightly from the current 
number. In mode 2, interpolated randonmess is used, where a random number is 
averaged with the current number, and thus more closely related to the current 
number than a purcly random number. Finally, in mode 3, a non-interpolated, 
completely random number that is not related to the current valué is used. Each 
paranreter is then modified by the valué resulting from the current mode. Both 
mode 0 and mode 1 are utilized to update the panning amongst the 16 speakers. 
mode 1 is used to update the amplitude, band-pass frequency, and EQ bandwidth. 
The LFO frequency is updated in mode 2. The sequencer runs indefinitely; the du- 
ration of the output frequencies are contoured proportionally using a meandering 
Fibonacci series that creates asymmetrical cycles. The duration of each note is 
also further adjusted using Brownian randonmess (mode 1). In using this mode to 
update the majority of parameters serves to create the perception of naturalistic 
sounds. The output frequency is updated via a more complex amalgamation of 
the outputs of all modes, and then passed on to the Bark scaling unit. 

The Bark scale is a non-linear frequency scale that was psycho-acoustically 
designed originally by the acoustics scientist Eberhard Zwicker in order to trans¬ 
íate frequencies into valúes that sound perceptually linear (to human ears). The 
mapping from an input frequency to a corresponding output valué in the Bark 
scale is governed by the following equation: 

b = 12 ían -1 (0.00076/) + 3.5 tan~ l (f/ 7500) 2 (3.10) 

Here / is an input frequency valué and b is the output Bark number. The non- 
linear frequency response of human ears is described by the concept of critical 
banding. The width of each of these critical bands remains more or less constant 
up to 500 Hz, and then jumps by approximately 20 percent thereafter [244], The 
Bark scale is a sequence of these critical bands that are discretely enumerated. It 
is utilized as a compositional tool in order to generate a perceptually linear sweep 
of pitches and to define an evenly distributed pointillistic texture, and to create 
a solid mass of sounds. 
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3.6.2 Multi-modal collaboration research 

Annular Genealogy is an innovative interactive composition that invokes a synaes- 
thetic experience by emphasizing the use of synchretic feedback on múltiple levels, 
the physical, the digital, the interconnected, and the human. Physical feedback 
occurs as multimedia information is put into the performance space and then 
recaptnred by physical sensors. For instance, this happens continuously as the 
contact microphones reintroduce the output signal as an input signal into the 
composition engine. Digital feedback occurs internally in the software engines as 
output data is immcdiately used to update aspects of the digital heuristics. In 
the visualization engine, this occurs frarne by frame as the previously generated 
texture is used as the new input texture to be distorted by the fluid vectors. Inter- 
connected feedback occurs when the output data is sent over a wireless network 
to external software processes. This included the fluid vectors created on the iPad 
updating the stochastic seqnencer running in SuperCollider on the laptop, and the 
sequencing data influencing the image processing parameters. Human feedback 
occurs as the performers use their instruments to influence the software, via multi- 
touch and live coding. The original interactive performance was shown at the Bits 
& Pieces media arts exhibition in Santa Barbara, California. The trickiest part 
in developing the original composition was to find an appropriate balance of auto- 
matically generated composition/visual via feedback and performers input. That 
is, it took time to find an aesthetic balance between the human and the computa- 
tional. The iterative generative systern can quickly fall into patterns that becorne 
either repetitive or overly chaotic. Finding the Creative “cusps” teetering between 
these two extremes was the rnost rewarding aspect of the performance. Whilc 
sorne of the results of interconnecting múltiple feedback layers are unpredictable, 
the performers nonetheless begin to have an intuition as to how their actions will 
update the overall composition. For example, while there is no direct mapping of 
how the visualization data will update the compositional structures, after sorne 
experience using the iPad interface, it becomes clear that certain gestares during 
certain kinds of passages generate a particular shaping of the composition. We 
also found it interesting to re-conceive the performers role as a “guider” of aes- 
thetics, rather than as a creator. A possible direction for future versions of the 
picce would be to more explicitly highlight the effect that an interaction has as 
it is transmuted from one médium to the other. Although the focus of the piece 
has been on interconnecting different feedback loops, one obvious feedback cycle 
that we did not attended to is the incorporation of the audience as a co-performer. 
Though adding extra elements to the piece increases the difficulty of maintaining 
aesthetic balances in the visual and aural domain, a careful integration of audi- 
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ence interaction in future versions of the piece will incorpórate audience members 
as participants rather than as passive viewers. 

3.7 Conclusión 

In this chapter I explored the various iterations of an research project, Fluid Au¬ 
tómata, that liad both artistic and research outputs. This example was used 
to show how a single project can evolve through múltiple implcmentations and 
through the reframing of the original intentions and ideas. These new ideas, I 
contend, would not have occurred without undertaking this process of implemen- 
tation. That is, through implementation I was able to generate new research foci 
as wcll to see new ways to incorpórate the core technology into new art installa- 
tions. 

Although the Fluid Autómata Project is not an art-science project per se, the 
g eneration tasks I undertook I belicve are useful for collaborative interdisciplinary 
projects. Ruth West, in her recent paper, “DataRemix: Designing The Data- 
made Through ArtScience Collaboration,” explains her bclicf that media artists 
can develop approaches that “complcment empirical methods being developed in 
visualization, information visualization, visual analytics, statistics and machine 
learning.” Specifically, she posits three “meaningful” aspects of artistic practice: 
“the externalization of intuition”; “the creation and use of metaphor”; and the 
Creative use of modalities that “make the abstract experiential” [227]. While it 
is clear that many types of art projects involve solving a host of technical and 
aesthetic problems, it is not as common to think of art projects as generators of 
worthwhile research questions outside the domain of media arts itself. But in fact 
art projects are precisely the appropriate location for both generating interest- 
ing research questions and exploring the boundaries of the questions, of finding 
out where known techniques are no longer sufficient and new research must be 
conducted. In fact, media artists often bounce back and forth between Creative 
exploration and research. The exploration leads the artist to unsolved problems, 
at which point they must switch gears and solve those problems. Of course, this 
requires that artists have the knowledge/skills to make these leap. On the one 
hand, they are intimately engaged with what the issues are. On the other hand, 
the artist may not have the appropriate skill set or enough time to conduct the 
research on their own. In the next chapter, I discuss larger collaborative research 
projects involving myself and interdisciplinary teams of researchers with different 
expertise, and discuss how one role of media arts is to function as an explicit 
mediation between múltiple perspectives and domains. 


86 



Chapter 4 

Múltiple Immersions: 
Collaborative Tools for 
Interdisciplinary Research 
Projects 


Design with low thresholds, high ceilings, and wide walls... 

- fro7n “Creativity Support Tools,” Ben Shneiderman [195] 


4.1 Introduction 

The previous chapters introduce art projects that have foci that parallels research 
in interaction and visualization, and also describe in detail a hybrid art and re¬ 
search project that explored aesthetic possibilities as wcll as questions of interac¬ 
tion, simulation, and collaboration. More recently, I have become more focused on 
working on a range of interdisciplinary projects that involve input frorn different 
dornain experts and inelude a focus on design and usability as a component of facil- 
itating research. These inelude projects with researchers in genetics, in cognitive 
Science, in health informatics, in machine lcarning, and in natural language Pro¬ 
cessing. In particular, I look in detail at two of these projects, Natural Materials 
Browser and Augmented Communication Tools. These two projects do not have 
a direct artistic focus, although they are still concerned with usability and design 
issues. In addition to showeasing these projects as examplcs which highlight the 
methodological approach of augmentation - of enabling research through novel 
interaction techniques and representational techniques - they also serve to intro¬ 
duce the idea of mediation. I believe that the success of these projects was due to 
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having an interdisciplinary perspective that enabled effective communication and 
workflow. The media arts are advantageous to have in interdisciplinary collabora- 
tions because media artists already opérate in a hybrid space between disciplines, 
and they are already adept at bridging different concepts and representations and 
mediating between different languages. In the sections below, I explore how the 
methodological themes introdnced in this dissertation enable these projects to be 
more effective. 

4.2 The Natural Materials Browser 

The Natural Materials Browser is a project conceived in collaboration with the ma- 
terials scientist, Tony Fast [70]. We wanted to find a way to support the Materials 
Genomic Initiative by making it easy to share volumetric data sets. In particu¬ 
lar, we wanted to be ablc to load in múltiple high-resolution material samples to 
support analysis and comparison tasks. In addition to being able to share data, 
we wanted to be able to enable materials scientists to conduct statistical analyses 
of their data. The project first was discussed after Tony Fast saw an installation 
called New Dunites , lcd by Andrés Burbano and Danny Bazo, where I developed 
volume-rendering software to visualize data from ground-penetrating radar that 
explored the ruins of the Ten Commandments in dunes in central California [20]. 

In materials Science, the development of new materials with enhanced perfor¬ 
mance requires a detailed understanding of the internal structure of existing ma¬ 
terials to connect the causal relationships between structure and property. Sorne 
researchers are interested in discovering how the organization of the internal struc¬ 
ture, or microstructure, of natural materials lead to their enhanced structural 
properties and, subsequently, how to artificially recreate these same conditions. 
Materials have complex 3D structures and with emerging characterization tech- 
niques (e.g. atornic probe microstructure [190], micro-computed tomography [12], 
serial sectioning [54]) materials scientists are capablc of producing detailed volu¬ 
metric dataseis from a breadth of materials Systems. Visualizing the 3D structure 
of materials is complicated due to the wide array of material features (e.g. com- 
position, orientation) whose 3D arrangement directly effects the materials perfor¬ 
mance. Even in the case of binary phase materials, the topology of the material 
is rich and complex, as indicated in the data used for this study. These exarn- 
plc dataseis are generated by micro-computed tomography (micro-CT) of various 
nutshclls - pecan, hazelnut, coconut, and macadamia - with salient pirases that 
are pore and solid. Micro-CT produces voxel-based 3D data with each voxcl being 
described as phase zero or one, pore or solid, respectivcly. By visualizing the com¬ 
plex pore structure of natural materials, materials scientists hope to understand 
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the source of their exceptional structural properties via both the raw data and 
subsequent statistical analysis. 

The Natural Materials Browser is prototype tablet application that aids the 
materials scientist in the preliminary analysis of volumetric dataseis and offers 
the following benefits: it is low-cost and easy to deploy; it allows the user to view 
the data from arbitrary angles and positions; it provides múltiple visualizations of 
the volumetric samples; it couples real-time statistics with the visualizations; and 
it provides a sense of immersion within the data. The Natural Materials Browser 
airns to augment the existing workflow of the materials scientist by providing 
an easy way to discover structural and statistical properties of material samples. 
By integrating existing visualization and interaction techniques, our application 
effectively provides insight about volumetric dataseis to materials scientists that 
would otherwise only be noticed through more extensive investigation. 

The main viewport of the application acts as a window that displays a portion 
of the currently loaded sample. Rotating the tablet updates the view in real-time, 
and provides a new perspective of the data from a different angle, created via 
interpolating between the original 2D image slices of micro-CT data. Additional 
viewports simultaneously provide further information regarding the current view 
and the sample as a whole, including: a geometrical interpretation created using a 
mesh of an isosurface of the scalar data, a view of the data using a GPU raycasting 
technique, and a statistical representation of the data (using autocorrelation). An 
interactive legend situates the user by highlighting the user’s location within the 
volumetric data. The user can swipe the screen to load in another microstructure 
dataset, encouraging the user to make comparisons between the different materials. 
Figure 4.1 shows a screen capture of the tablet application with the statistics 
viewport, the interactive legend, and the raycasting visualization visible along the 
bottom part the screen. 

A host of visualization techniques exist to aid in reasoning about volumetric 
data. More general Solutions for visualizing material data samples are discussed 
in [194], Material Science dataseis can be visualized with high-end desktop soft¬ 
ware packages such as Avizo Fire 1 and the Accelrys Materials Studio 2 , and with 
frameworks built using the VTK visualization toolkit [186]. For instance, [240] 
describes an “image-to-simulation” framework that makes use of múltiple visu¬ 
alization techniques to enhance and validate models describing the structure of 
porous rock. However, in practice the most common methods of preliminary anal¬ 
ysis usually involve viewing the raw data as a series of discrete image slices and 
generating statistical plots for each slice, often independently of each other. The 

1 http://www.vsg3d.com/avizo/fire 

2 http://accelrys.com 
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Figure 4.1: A screen capture of the Natural Materials Browser. 


maiu issue witli this approach is that the data sample is ouly viewed from a sin¬ 
gle direction and decontextualized from the full dataset. Furthermore, the visual 
inspection of the data is often decoupled from the statistical analysis of the data. 

On the other end of spectrum, extensive work has been done on placing vol- 
umetric data in virtual reality (VR) environments in order to provide enhanced 
immersion and greater exploratory possibilities. An early report on the opportuni- 
ties and challcnges in using VR for scientific visualizaron purposes describes the 
importance of creating an immersive experience that lets the user to feel as if they 
are interacting with the data itself, rather than a picture of the data [19]. Another 
early study attempts to quantify the importance of immersion and concludes that 
users of VR applications have a better mental frame-of-reference of the data than 
they would using normal desktop applications [166]. More recently, [121] finds 
that there are significant benefits to analyzing volumetric data in immersive en¬ 
vironments, and moreover concludes that the benefits exist for certain tasks even 
when only sorne aspects of immersion are present. [213] provides an overview 
of different uses of VR environments to aid in scientific understanding and em- 
phasizes the possibilities available via immersive experiences, but also notes the 
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disadvantages in cost and accessibility of using full-scale VR environments. For 
the basic needs of preliminary data analysis, fu.ll scale VR techniques may be im- 
practical, requiring extensive preparations and expensive equipment. To that end, 
[35] describes a low-cost VR project that uses a multitouch table to interrógate 
volumetric data that is shown on a stereographic display. 

The application forgoes some aspects of common VR elements, such as stere¬ 
ographic rendering and head tracking, but nonetheless aims to create a sense of 
immersion within a virtual environment. Using the gyroscope and multitouch 
sensors on the tablet Computer, it simulates the movement of the user through 
the volumetric dataset. In some sense, our application is an untethered versión of 
earlicr projects that use a palmtop or mounted displays on a mechanical arm to 
navigate “3D-situated information space” [199, 65]. Although there is conflicting 
evidence in the literature regarding the importance of accurately tracking motion 
for effective immersion (cf. [180, 181]), a recent paper concludes that ”allowing 
for full-body rotations without actual walking can provide considerable perfor¬ 
mance benefits, even for complex and cognitively demanding navigation tasks” 
[175]. Magic lens interfaces are frequently used in volume visualization tasks. For 
instance, [174] describes an application that uses a magic lens interface as a vir¬ 
tual “flashlight,” highlighting fiber tracts in volumetric medical imagery. Another 
recent paper compares mouse interaction with gesture-based magic lens interac- 
tion and finds that magic lens interaction outperforms mouse interaction in time 
and accuracy for orientation matching tasks [116]. Our application provides a 
low-cost mechanism that lielps users build an effective mental model of the 3D 
dataset through the use of a magic lens interface and full-body rotation gestures. 



Figure 4.2: A user changes his vantage point of the currently loaded dataset by 
physically rotating inside the virtual space. 
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4.2.1 Elements of the Natural Materials Browser 

The main interaction metaphor of the Natural Materials Browser involves placing 
the user within the meso-scale volumetric data sarnple and enabling the user to 
freely investígate qualities of the microstructnre via physical rotation and multi- 
touch gestnres. Using the tablet device as a “magic lens,” the user rotates within 
the snrronnding data to change his or her perspective and uses a multitouch ges- 
ture (pinching) to move forward or backward along the current viewing anglc. 
(Figure 4.2 shows an example of a user navigating the virtual volunte using the 
iPad.) Other visualizaron modules are placed in additional viewports to display 
different interpretations of the data, including a spatial statistics analysis. These 
viewports are also npdated in real-time, providing a synergistic experience of the 
virtualized data. The user can also quickly load in other dataseis, which has 
the potential to aid comparative investigations of different materials (or different 
samples of the same material). The Natural Materials Browser was developed 
for a third-generation iPad device using a custom C++ framework and custom 
OpenGL ES 2.0 shader programs. The example dataseis used for development 
and testing are each 512 3 voxels. 

4.2.1.1 Interpolated Viewports 

Despite the richness of 3D material Science dataseis, their visualization is often 
restricted to a single reference frarne oriented in a single direction. That is, the 
material is primarily visualized as a collcction of orthogonal 2D slices. This ap- 
proach to visualization restricts the scope of the data that can be visualized whilc 
providing an incomplete engagement with the data. The main viewport of the 
Natural Materials Browser instead displays the interpolated slice across the 3D 
volume at a particular position and orientation, created in real-time as the user 
repositions and interacts with the tablet. The coloring of the main viewport is 
based on sampling the data at three points per pixel: the intersection of the view¬ 
ing slice with the material (green), and two points orthogonal to the viewing slice, 
one slightly bchind (red) and one slightly in front of (blue). This coloring gives an 
indication of the strnctnre of the material at the current viewing slice, and also 
how particular fibers change (i.e., shrink, expand, and/or twist off in different an- 
gles) as the user moves forward and backward along the current direction (using 
the pinching gesture). Since the OpenGL ES 2.0 specifications does not enable 
the use of 3D textures, the application uses a custom GLSL shader program that 
stitches together a 3D volume out of a series of 2D images. A second viewport 
functions as a legend, orienting the user by indicating the current viewing slice 
within the bounding box of the full dataset. The user can rotate the legend inde- 


92 



Chapter 4. Múltiple Immersions 


pendently, without changing the current view, in order to see their position more 
clearly (which might be necessary, for instance, when the current slice is nearly 
perpendicular to the viewer, and thus hard to see). 



Figure 4.3: A detail of the isosurfaces viewport when the hazelnut sample is loaded. 

The Natural Materials Browser has a modular design that allows for the in¬ 
clusión of other visualizations created using the same interpolated data. In our 
prototype, the user can select either an isosurface visualization or a raycasting 
visualization (in the lower right hand viewport of the application). The former 
uses a geometric mesh created by generating an isosurface from the data. The 
application uses a simple Phong shading witli múltiple light sources and color in- 
terpolation (where each colored light is mapped to a particular córner of the vol- 
umetric cube) in order to accentuate depth cues and highlight structural density. 
Currently, different colored lights are placed surrounding the different corners of 
the “virtual specimen” with the assumption that the different colors will help ori- 
ent the user as they rotate the interface. The latter uses a raycasting technique 
(similar to the volume rendering rnethod described in [36]) in order to emphasize 
the structure of the material along a greater distance range. Volume rendering 
techniques generally use a transfer function to indicate the color of particular 
densities at different colors. However, since the micro-CT data for the nutshell 
microstructures sets llave only binary density, a rainbow color map is used to 
indicate depth. In particular, structures that intersect the current viewing plañe 
are given a white color, but structures that do not are instead given RGB color- 
ing (where red indicates closeness and blue indicates distance). Opacity is used 
to indicate the density of structures along the viewing ray. Future versions will 
give users more control over lighting and coloring parameters. Figure 4.3 shows a 
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detail of the isosurfaces viewport; Figure 4.4 and the lower right hand section of 
Figure 4.1 show the raycasting viewport. 



Figure 4.4: A detail of the raycasting render on a current viewing slice. 


4.2.1.2 Statistical Interpretation 

Effective statistical metrics (e.g., volurne fraction, surface area) of the materials 
structure are the most prominent statistical descriptors used throughout the ma¬ 
terials Science community. However, these metrics fail to provide any information 
about the complex 3D topology of the material. In an effort to best leverage the 
information contained within rich, costly volumetric dataseis, a growing portion of 
the materials Science community is beginning to employ the use of cross-correlation 
statistics as material descriptors [80, 158, 109]. These metrics provide a statistical 
quantification of the material topology, and furthermore, they may encompass the 
more connnonly used, effective statistical metrics (which are often a subset of the 
cross-correlation statistics). 

The cross-correlation function quantifies the probability of certain material 
features existing at the liead and tail of a vector within a particular realization 
of the volumetric data. As an example, the cross-correlation of pore voxels and 
void voxels for a given vector indicates the probability of finding a pore voxel at 
the head and a void voxel at the tail of said vector. A special case of the cross- 
correlation is the autocorrelation, which interrogates the probability of finding 
like pirases at the head and the tail of vector (pore-pore correlations or void-void 
correlations). The autocorrelation function is the collection of all vectors possible 
in the volumetric domain; it is a convolution between two volumetric inrages that, 
with the assistance of the GPU, can be conrputed in real-tinre. 
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Figure 4.5: A detail of the statistics viewport showing a visualization of the 
autocorrelation function on a current viewing slice. 


Correlation functions have proved valuable in a breadth of materials Science 
problems, including image reconstruction [80], collation of dataseis [109], and 
evaluating structure uncertainty [158]. Visualizing the statistics also allows one 
to readily notice anisotropic properties in the structure that may be rnissed if 
only viewing the image itself. Despite these efforts and benefits, the use of spatial 
statistics in materials Science is still not commonplace, and providing a tool for 
real-time visualization of correlation functions will assist in its acceptance and 
application amongst the materials Science community. Figure 4.5 shows a detail 
of the representation of the autocorrelation function. 

4.2.2 Evaluation of the Natural Materials Browser 

The prototype application was evaluated via running cognitive walkthroughs with 
ten domain scientists. These domain scientists included five professors at three 
different universities and five post-docs at two different universities. The easy- 
to-deploy nature of the browser greatly helped the evaluation process and acces- 
sibility across different institutions. Since these scientists regularly examine and 
conduct statistical analyses of volumetric data, it was important to solicit their 
feedback on the application while they performed various preliminary data anal- 
ysis tasks, such as identifying interesting aspects of the materials and deciding 
whether or not the samples warranted more detailed investigation. Tony Fast 
and I were mainly interested in determining the potential valué of introducing 
the application as another element within the workflow of a materials scientist, 
rather than in the detailed measuring of the individual effectiveness of particular 
techniques. That being said, we were also interested in observing the responses to 
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various specific interaction and visualization aspects of the application, such as: 
whether or not the ability to navigate through an interpolated view of the data 
(via physical rotation and multitouch gestures) would be useful or distracting; if 
the real-time statistics viewport was a useful supplement for understanding cross- 
correlation functions; if the alternative views (raycasting and isosurfaces) provided 
any insights about the data; and if the simultaneous display of the the múltiple 
viewports was beneficial. Our cognitive walkthroughs were led by one of the au- 
thors (who is himsclf a materials scientist) well-aware of the general strategies 
used by scientists to familiarize themselves with a material dataset. He began by 
giving a brief demonstration of the Natural Material Browser , and then allowing 
the subject to interact with the application on his or her own. This was followed 
by specific, neutrally-phrased questions about various aspects of the application 
as well as more open-ended questions that allowed the subject to comment on any 
aspect of the application or to discuss aspects of his or her current and/or ideal 
workflow when first encountering a new materials Science dataset. 

Each of the domain experts indicated that they found the Natural Materials 
Browser to be a promising tool for investigating materials data. They were partic- 
ularly intrigued by the idea of easily viewing volumetric data at arbitrary viewing 
angles. Overall, they also found the simultaneous viewports to be valuablc in com- 
municating the idea of correlation when seen directly alongside the corresponding 
interpolated view of the raw data. Similarly, the users became clearly aware of 
sorne qualitative features of the data from the statistics, such as the preferential 
orientation of pores when the material is looked at in different viewing planes. 
That is, simply by rotating the iPad around the virtual specimen, properties of 
the data were displayed that otherwise would have only been noticed with more 
extensive analysis. Some subjects questioned the valué of the raycasting and iso¬ 
surfaces visualizations and were initially confused by the color mapping. At the 
same time, some of the subjects recognized the potential usefulness of incorporat- 
ing additional viewport modules. Future versión of the Natural Material Browser 
will provi de altérnate visualization and quantification tools. In our sessions with 
the materials scientists we also discussed issues of ergonomics, such as the poten¬ 
tial for fatigue that might occur from full-body rotations during extended use of 
the application and whether or not it was awkward to use one hand to hold the 
tablet whilc the other hand was interacting with or pinching the tablet. Since the 
application is envisioned as a “browser” - a first pass that provides enhanced ini- 
tial information about the statistics and anisotropy to indicate intriguing aspects 
of the materials data - materials scientists will not necessarily use the tablet for a 
long period of time, but rather take the information they have gathered from our 
application to conduct further analysis on a more powerful workstation. That is, 
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the application will not replace any component of the materials scientist’s work- 
flow, but rather will augment the workflow by provicling a helpful initial overview 
of the data. 

The Natural Materials Browser is an immersive, interactive, real-time tablet 
application with múltiple viewports for the simultaneous visualization and statis- 
tical analysis of a volumetric dataset that augments the workflow of the materials 
scientist. While the initial feedback from dornain scientists is encouraging, further 
work on evalnating the efñcacy of the immersive aspects of the application needs 
to be carried out. In particular, we plan to carry out a more rigorous comparison 
with an implementation that does not require physical rotation (e.g., rotating the 
data using a mouse or virtual trackball). Furthermore, we would like to extend 
the modularity of the application so that a wider range of visualization and anal¬ 
ysis techniques could be selected as needed for particular dataseis. One initial 
design goal of the application was to facilitate comparisons between dataseis, and 
while the user cnrrently is able to switch between the pre-loaded dataseis, no di- 
rect methods for visualizing differences is available. In part, this is a limitation 
of the screen size of the iPad. A futnre goal is to investígate ways in which the 
Natural Materials Browser could be used to internet with a full-scale VR System 
featnring larger stereographic displays that could be more appropriate for compar- 
ative tasks. Finally, based on the feedback from our discussion with the expert 
users dnring our evaluation, we also plan to explore the effectiveness of different 
coloring schemes for the main viewport as wcll the visualization modules. 

Materials Science is a largely collaborative field that requires the varied skills 
of many experts to design and deploy new materials. These efforts often generate 
large collcctions of dataseis. Facilitating the transfer of this data amongst collab¬ 
orative materials Science efforts stands at the forefront of national directorates, 
such as the Materials Genome Initiative [206], which airns to create the cyber- 
infrastructure capablc of this data distribntion. The Natural Materials Browser 
provides a low-cost, readily available tool for analyzing materials Science data, 
and could be extended to enable end-nsers connected within a workgroup in the 
cyberinfrastrnctnre to visualize, annotate, share, and discuss data in a Progressive, 
collaborative manner. 

The Natural Materials Browser is an examplc of a project that was created 
out of media arts focus on novelty of representation and interaction, but that was 
only able to be realized in collaboration with a dornain expert in materials Science. 
While the project clearly serves to augment the existing workflow of a material 
scientist, we hope that by making it easier to download and compare dataseis, 
and then to statistically analyze them, that it will generate new research ideas 
for the materials scientist as well. Although I have not explored this in depth, 
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I bclieve that the materials dataseis could generate an artistic output as well, 
as they introduce a range of naturalistic, meso-scale patterns and structure not 
otherwise availablc to be observed or represented. 

A first step in working with someone in a different field is understanding their 
language. To develop the Natural Materials Browser, I worked closely with Tony 
Fast, and liad to learn basic tenets of materials Science, as well as specific aspects 
of his current areas of research. Likewise, I liad to explain some of the possibilities 
and limitations of the iPad platform that we were using to develop the application. 
The Natural Materials Browser is an example of an interdisciplinary project well 
served by the methodological themes and their associated tasks. However, both 
Tony Fast and 1 were computationally savvy and both interested in visualization. 
1 bclieve that projects between participants who have less similar perspectives 
are a better case study for these methodological themes, especially the theme of 
mediation. When múltiple peoplc are involved, as in the Augmented Communi- 
cation Tools project, described next, the issue of communication becomes even 
more important. In addition to having to learn enough about health informatics 
and natural language processing, I found that another skill that was hclpful to the 
success of the project (and writing the grant) was being able to serve as a kind of 
mediator, or translator, between domain experts. 

4.3 Augmented Communication Tools 

Another ongoing project also involves collaboration with domain-experts from dif¬ 
ferent fields. I am currently a co-PI on a grant with a data scientist focused on 
natural language processing (NLP) and a nursing professor concerned with issues 
in health informatics. Our project, Augmented Communication Tools , seeks to im- 
prove methods of communication between nurses. Specifically, we are developing 
a machine learning System for automatically identifying high-risk clinical events 
and presenting this information with novel visualization tools to more effectively 
transmit knowledge between the nurses [78]. 

The introduction and subsequent adoption of the Electronic Health Record 
(hereafter, EHR) is a step forward in automating data collection useful in the 
analysis of patient health and prevention of adverse health events [31]. How¬ 
ever, unintended consequences mitigate the effectiveness of the EHR [29, 92, 203]. 
Most problematically, nurses alternately taking care of a patient are unable to 
effectively sift through the large amount of data availablc via the EHR in order 
to find pertinent information. Studies of nurse behavior has found that many 
nurses make an effort to talk to each other face-to-face as they change shifts [27]. 
Ideally, these face-to-face conversations provide a way for the responding nurse 
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(leaving the current shift) to explain his or her interpretation of the patient’s well 
being to the receiving nurse (starting the next shift). The absence of this clialog 
may account for an alarming number of miscommunications that have lead to 
catastrophic events in patient health [203]. For instance, a report by the Insti- 
tute of Medicine [117] finds that up to 98,000 patients che per year as a result of 
complications of therapy due to ineffective communication and, moreover, that 
errors in communication cost US hospitals an estimated $12 billion annually [56]. 
Although providing verbal summarizations of patient health during in the hand- 
off between responding and receiving nurses can be helpful, these summarizations 
can themselves be misinterpreted or ignored [28]. 

The Augmented Communication Tools project involves the development of an 
interactive visualization systern that augments EHRs to improve nurse-to-nurse 
communication. Our systern uses a novel application of machine lcarning and 
natural language processing techniques to generate a series of potential clinical 
events [156] and, furthermore, can offer reasons why these events are plausible 
based on an analysis of: the vital signs recorded in the EHR; the verbal “hand- 
off” summary of patient health rnade by the responding nurse; and historical 
EHR data of patients that liad unexpected adverse clinical events. In particular, 
we look at data related to six categories of clinical events that are most likcly 
to be a precursor to unexpected patient death: uncontrollcd pain, sudden fever, 
bleeding, changes in respiratory status, changes in level of consciousness, and 
changes in output. We further describe a novel, interactive shift-summarization 
visualization tool that provides: 1) the automatic proposal and notification of 
these potentially adverse clinical events; 2) a way for nurses to verify the likelihood 
of these events; 3) a mechanism for annotating the automatic proposals and for 
allowing the nurse to propose their own events; 4) and finally, an interactive 
tool that lets the responding nurse associate vital signs as they evolve over time 
with an overview narrative, making it easier to inclicate highly pertinent data 
to the receiving nurse. In effect, our systern airns to replicate, to sorne extent, 
aspects of face-to-face nurse communication that may have been lost through the 
introduction of EHRs, and to formalize the verbal summarization that sorne nurses 
have improvised. That is, our visualization systern aims to make it easy for nurses 
to create, transmit, and verify narratives about patient health that augment their 
expert knowledge for improved decisión making. 

Both the development of a computational model to predict clinical events 
and the creation of interactive visualization tools based upon this model make 
use of data gathered from real-world scenarios where EHRs are used by nurses 
in making decisions regarding patient treatment. Currently, we have extensively 
annotated data generated through interviews with 37 nurses using EHRs who 
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oversaw patients who died unexpectedly, and we are streamlining the interviewing 
process in order to gather 100 more samples of nurse (mis)communication [57]. 
Although this work is ongoing, it brings together interdisciplinary expertise in 
liealth informatics, machine learning, and text analytics with the aim of creating 
an effective interactive visualization tool with the potential to save lives. 

4.3.1 Generating and Explaining Clinical Events 

The computational modeling component of onr visualization is divided into four 
interrelated modules, each making use of information available in the EHR and/or 
the transcriptions of verbal summarizations, or “hand-off” reports, created by the 
responding nurse. The first two models provide a detection facility for clinical 
events, as well as a probabilistic outcome prediction mechanism to determine the 
relative likclihood of an undesired outcome given the detection of a clinical event. 
To make certain that the hand-off report is properly connected with the EHR 
data, a third model grounds the narrative of the hand-off report with vital sign 
measurements in the EHR. These links between the two content modalities are 
further exploited in the fourth model, which generates an automated summary 
of the hand-off report, highlighting information that is predictive of a negative 
patient outcome. 



Figure 4.6: A prototype of the overview page of our shift-summarization tool. At 
the top we see a simple bar chart indicating high-risk clinical events (determined 
through an evaluation of historical EHR data). At the bottom we see a list of 
events generated by our system (based on expert rules), alongside related textual 
snippets from the nurse’s “hand-off” report. 
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Generated Clinical Events 

Thrombosis (32%) 

- high variability of heartbeat 

- low blood press u re 

- insomnia 

"patient was not sleeping" 

"pressure seemed unusually low" 

"weird fluctuation in heart rate" 

SleeD Apnea (26%) 


- insomnia 

"patient was not sleeping" 

- difficulty breathing 

"pressure seemed unusually low” 

- high amount of movement 

"weird fluctuation in heart rate" 

- low blood pressure 

"lots of drool" 

"tossing and turning" 

"muttered to himself" 


Figure 4.7: A prototype of the overview page of our shift-summarization tool. At 
the top we see a simple bar chart indicating high-risk clinical events (determined 
through an evaluation of historical EHR data). At the bottom we see a list of 
events generated by our system (based on expert rules), alongside related textual 
snippets from the nurse’s “hand-off” report. 


4.3.2 Visualization Tools for EHRs 

The existing design of EHRs are graphically cluttered and parsing patient data can 
be cognitively-taxing, even for experts [224]. Moreover, in the attempt to provide 
information about all possible indicators of clinical events, ironically, the most 
relevant and potentially life-threatening of these can be obfuscated [28]. While 
previous work has explored different approaches to visualizing and managing the 
complexity of EHRs, for example [3, 169], our shift-summarization tool provides 
enhancements to EHRs to improve nurse comprehension of and communication 
about patient data that is related in particular to signáis of a sudden and unex- 
pected change in patient condition. The primary components of onr visualization 
enhancements are based directly on the data generated by the compntational 
models: the alignment between EHR data and the responding nurse’s verbal sum- 
marization of the events during the shift; a series of generated explanations as 
to why these events happened; and an overview likelihood, based on historical 
data, of whether or not these generated events could lead to adverse changes in 
patient condition. Additionally, we incorpórate temporal aspects of the EHR 
data in order to augment the receiving nurse’s decisión making when verifying the 
recommendations produced by the data analysis module. 
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Specifically, our visualization enhancements provide: a way to notify nurses 
about high-risk situations; detailed information that nurses can use to verify these 
notifications; a way for nurses to meaningfully annotate their interpretaron of 
the patient health in relation to inferred clinical events; and finally, a System 
that encourages a nurse to link annotations to temporal data in order to nárrate 
the overall story of how the patient’s health evolved over the course of a shift. 
Each enhancement builds on the previous, adding functionality that increases the 
effective transfer of pertinent information between nurses when using EHRs. We 
are implementing the prototype of our shift summarization tool on an iPad tablet, 
but expect also to port it to desktop environments or other mobile devices. 

4.3.2.1 Enhancement 1: Shift Summarization (Notify and verify) 

Our primary enhancement clcarly presents the results of our computational anal- 
ysis. All six of the major clinical events are listed, and high-likelihood events (as 
determined via a historical analysis of the EHRs of patients who suffered these 
events) are highlighted. The receiving nurse is thus, at a glance, informed as to 
whether or not the patient is in inuninent danger. 

In addition to this high-level overview specific to the major clinical events, we 
also generate a list of other inferred clinical events that occurred during the shift 
and provide the evidence as to why they were inferred. That is, for each of the 
generated clinical events, we link to the EHR monitoring data or the transcription 
of the nurse’s verbal summarization that led our System to conclude that the event 
occurred. 

Explicitly presenting data that shows the logic behind the computational 
inference functions both as a way to create trust in the System, as has been 
shown in research on (or implemented in) recommendation Systems, such as 
[34, 41, 77, 172, 197], and also provides a starting point for verifying or inval- 
idating the automatic notifications, which could be especially important in the 
case of false alarrns. A third component of this first enhancement is to provide an 
interface for browsing the alignment between EHR data and text. This enables the 
nurses to verify results of our automated system easily, and also to search freely 
for patient-specific information that could bolster or invalídate the automated rec- 
ommendations. Figures 4.6 and 4.7 show a prototype of this enhancement. At the 
top we see a simple bar chart indicating high-risk clinical events. At the bottom 
we see a list of events generated by our system, alongside related textual snippets 
from the nurse’s hand-off report. Figure 4.8 shows an example of how a nurse 
selecting a particular explanation for a generated clinical event can instantly see 
more detail about the aligned text and EHR vital signs that lead our system to 
present this explanation. 
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4.3.2.2 Enhancement 2: Issue Tracking the EHR (Annotate) 

This enhancement aims to promote dialog between the nurses, to foster engage- 
ment with relevant information about patient health, and to provide accountabil- 
ity for the nurses caring for the patient. 

The generated clinical events are essentially an interpretation of the raw data 
and the nurse’s verbal summary of the shift. Moreover, the alignment process 
between the EHR data and the verbal summary is also based on encoded as- 
sumptions. We provide a systern for the nurses to agree or disagree with the 
automatically generated clinical events and to annotate them with additional per- 
tinent information. The annotations are in the form of a predefined comment- 
such as “agree”, “disagree”, or “inconclusive”- and, optionally, space is available 
for further detailed commentary. By providing a mechanism that operates, es¬ 
sentially, as an issue-tracker, nurses have an opportunity to create and respond 
to forums about particular events that may be important to the patient’s health. 
In issue-tracking software or websites used for software projects (e.g., GitHub 3 or 
Googlc Code 4 ), this type of commentary is used to build consensus on interpreta¬ 
tion, to expedite decisión rnaking, and to facilitate conversation [11], By requiring 
the nurses to annotate each of the generated events as well providing the ability 
to define their own, we instantly create a focused dialog about the relevant issues 
regarding the health of the patient. Furthermore, this mechanism creates a trail 
of accountability, as the nurse can explicitly explain their reasons for disregarding 
an event, or modify the reasoning bchind why the event occurred. 

The enhancement is split into two related components, one for the responding 
nurse and the other for the receiving nurse. First, we allow the responding nurse 
to evalúate the shift summarization created by Enhancement 1. In particular, we 
allow the nurse to annotate the generated events, either with predefined terrns, 
or with a more detailed textual explanation. Thus the receiving nurse has more 
information regarding consensus between the automated interpretation and the 
nurse’s interpretation of events in the shift. Second, we similarly provide a space to 
indicate agreement or disagreement with generated events and allow the receiving 
nurse to indicate that he or she has read and the responding nurse’s annotations 
as well as space to provide additional commentary. 

4.3.2.3 Enhancement 3: Telling a story via temporal data (Nárrate) 

This enhancement extends the annotation mechanisms described in Enhancement 
2, allowing the responding nurse to link their interpretation of the patient’s health 

3 https: // github.com 

4 http: //code. google. com 
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to particular events in time. That is, we allow the responding nurse to create a 
curated timeline of the patient’s health as it evolved over the course of the previous 
shift. The receiving nurse can then use this temporally-contextualized data to 
augment his or her decisión making process during the current shift. Previous 
research has investigated visual information seeking over temporal data across 
múltiple EHRs [6, 176, 223, 224], Our project emphasizes the temporal aspects 
of the patient data over the course of a single shift. This enhancement allows the 
responding nurse to browse the generated events and annotations along with the 
temporal EHR data (e.g. the “flow sheet”) and then to create a visual narrative 
of how the nurse reasons about the possibility of clinical events. Narratives are 
made up of pertinent, sequential events, and providing a System that allows the 
nurse to explain the events of the shift in a narrative format attempts to replace 
an important aspect of face-to-face communication that is otherwise lost in the 
hand-off. Figure 4.9 shows a sketch of the proposed enhancement in which an 
annotated timeline is associated with potential clinical events. The circles indicate 
highlighted raw vital signs frorn an EHR flow sheet that led the responding nurse 
to make a commcnt. 

4.3.3 Evaluation Methods and Considerations 

Our shift-summarization tool aims to clearly represent the automated interpreta- 
tion of patient data as well as the sentiment of the responding nurse and, second, 
to ensure that the receiving nurses trust the representation sufficiently to incor¬ 
pórate it into thcir decisión making. Our systern is rneant to augment (and not 
replace) nurses’ expert skills, and our evaluation is airned not only at justifying 
appropriate visualization methods but also to measure the effectiveness of their 
integration into current practices. 

The two rnain contributions throughout these enhancements are a) clearly 
representing the results generated by our computational models, and b) providing 
an interactive interface with which to support rapid exploration of EHR data. We 
hypothesize that each additional enhancement provicles increasingly more effective 
communication leading to more accurate diagnoses of at-risk patients. We further 
hypothesize that our systern increases the nurses’ ability reason about salient 
information, whilc simultaneously reducing the amount of time they spend filtering 
out irrelevant data. To test these hypotheses we plan to run user-studies with 
nursing students and practicing nurses. These user studies will examine whether 
an expert using the proposed enhancements performs better than the automated 
systern alone. We are also concerned about the level of trust an expert rnight 
have in the automated systern and also with the amount of time it takes to use the 
enhancements. Obviously if there is no trust in the systern or if it causes an undue 
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Thrombosis (32%) 

- high variability of heartbeat 

- low blood pressure 

Temp Sats 

BP 


44 

67 

48 

55 



32 


35 


... I noticed that the patient was not sleeping all that 
much. We gave the patient more pillows and replaced 
a flickering light bulb, but the patient still reported 
some insomnia. with lots of tossing and turning. I also 
noticed he was irritated and there were some 
occasions when I observed him muttering to himselt. 
When he did manage to sleep, he left lots of drool on 
his pillow. Also, despite an increased dosage of 
fludrocortison^MreaMii^rthoti^ivDOtension, the 
patient's I 

continué to monitor the weird fluctuation in heart rat( 
that doesnl seem to be going away ... 



Figure 4.8: An example of how a nurse selecting a particular explanation for a 
generated clinical event can instantly see more detail about the aligned text and 
EHR vital signs that lead our system to present this explanation. 


burden, then nurses will be less likely to use the visnalization tools in real-world 
situations. Thus, in addition to the user-studies, we will also conduct cognitive 
walkthroughs in order ascertain the ease of use and potential rate of adoption 
of the tools. As discussed in [155] and [195], the iterative design of information 
visnalization tools based on detailed feedback from cognitive walkthroughs with 
domain experts can be an effective way to create Systems that will be useful in 
real-world environments. 

While our initial results indicate the effectiveness of the relatively straight- 
forward mapping of our automatically generated clinical events to visual alerts, 
we need to more thoroughly evalúate whether the additional enhancements are 
worthwhile. Our hope is that the integration of heterogenous patient data cou- 
pled with novel visualization tools will be an effective way to transmit a narrative 
of patient health. Indeed, whether or not this narrative is perfectly accurate, it 
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Figure 4.9: An annotated timeline that helps to tell a story by relating pertinent 
points of data on particular vital sign streams to nurse interpretation. 


should adequately engage with and highlight the most relevant patient data so 
that another domain expert (i.e., the receiving nurse) can quickly validate (or in¬ 
valídate) it and thereby become aware of potential patient risk. Finally, we hope 
that this additional level of annotation (via the creation of the narrative and its 
validation) itself becomes data that can be taken advantage of in future analyses 
that promote accurate interpretation of and communication about patient health. 

4.3.4 Interdisciplinary Methodological Considerations 

The Augmented Communication Tools project would not exist without interdis¬ 
ciplinary collaboration between researchers with expertise in the fields of health 
informatics, natural language processing, and interactive visualization. Although 
1 was initially brought into the project to help design the interface- what possi- 
bly was considered a secondary concern of the project- through engaging with 
the methodological approaches outlined in Chapter 1, I was able to extend the 
aim of the project while enhancing the effectiveness of my participation in the 
project. Moreover, this project is allowing me to engage in fundamental research 
in information visualization, even as it serves as an application of existing design 
principies. 

The initial reason 1 was brought into the project was to augment the design 
of tools to support effective integration of EHRs into nurse communication. 1 de- 
signed a straightforward visual systern that would alert nurses as the results of the 
machine learning model which classifies patients accorcling to their risk for certain 
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outcomes. However, through a series of provocations , pointed questions aimed to 
tease out assumptions or flaws in reasoning, we realized that a straightforward 
representation of the machine learning recommendations would be problematic. 
Although we are convinced that the project, once complete, will be superior to 
current EHR tools, a central part of this project aróse out of thinking about 
why previous attempts with automated Systems have been less successful than 
expected. Through challenging sorne of our assumptions, we carne to realize that 
we were making flawed assumptions regarding the ability of our system to fully 
automate the decision-making powers and expertise of nurses. Through asking 
a series of questions about real-world use cases, we recognized that a better ap- 
proach would to create a system that allowed the nurses to guide the decisión 
making process themselves. We also realized that if the nurses did not trust the 
system enough to incorpórate it into their workflow then it would not matter how 
accurate our system was at predicting patient outcomes. 

An upshot of changing this focus was that the interface design, rather than 
straightforwardly representing the results of the machine learning system, became 
an integral component of the system. Instead of visualizing results, the interactive 
system lets the nurse navigate the underlying m.odel of the ML system and to un- 
derstand why it offers particular suggestions. Ideally, since our system has access 
to a large amount of historical patient data and also transcripts of nurse commu- 
nication, our system would normally make it clear to the nurse why a decisión 
was made. However, in many cases, the classification model would not capture 
the reasoning of an experienced nurse, and in that case, the nurse would still 
learn how their opinión cliverged from the expected conclusión. And, moreover, 
the nurse would be able to input that data back into the system, improving the 
underlying model for future use. This idea of using the information visualization 
system to visualize, explain, and enhance machine learning models for more effec- 
tive communication and analytics is, I believe, an exciting area for future research, 
and one that only emerged, or was generated, through the process of designing 
the system. 1 expect that during the process of implementation and testing the 
system further research avenues will also arise. 

In the case of this project, the interface itself serves as a kind of mediation 
both between the nurses and the automated system, and also between a nurse and 
the other nurses taking care of the same patient. On a more practical, day-to-day 
level, I often found myself working as a kind of a translator, summarizing the 
technical details of the machine learning system to the nursing professor who was 
less knowledgable about many aspects of Computer Science. I also was able to 
quickly understand what the nursing professor expected from the system, even 
when it was not initially clearly presented. I believe this ability to transíate 
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between different fields and to assist in the communication within a project stems 
directly from my experience as a media artist, used to working with novel, complex, 
or disparate information across disciplines on a regular basis. Two of the main 
tasks of mediation in collaborating inelude: summarization, or making sure that 
my understanding of the situation is accurate; and question asking, or asking 
questions that lead to simplifying a situation, rather than opening up the situation. 
This kind of question asking is of a different nature than the questioning task that 
is associated with provocation. Rather than using questions to expand the space 
of exploration, it instead served to focus our attention on particular issues and to 
prioritize the efforts of those involved in the project. 
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Conclusión &¿ Future Directions 


5.1 Summary of Contributions 

A primary motivation for this dissertation was the recognition that the roles of me¬ 
dia artists need to be more clearly defined and recognized as important to effective 
interdisciplinary projects, including those that engage in foundational research. 
As explored in the preceding chapters, I bclieve that media arts implcmentations 
directly enablc research in a variety of ways. Successful implementations are not 
just ornamental (althongh they can be); instead they promote new research. Fnr- 
thermore, media arts activities can directly support (and are not antithetical to) 
scientific methodologies. 1 also believe that provocation has an important place 
even in non-arts focused projects. By pointing out blind spots and by qnestion- 
ing assumptions, media artists also help expand the range of effective research 
and help rnake connection foundational research to cultural considerations. Fur- 
thermore, 1 contend that mediation is an important component of media arts 
activities and that mediation tasks contributed to by media artists are integral to 
art-science projects involving domain experts from múltiple disciplines. Bclow I 
recapitúlate the main content of the preceding chapters and list how they support 
these points. 

In Chapter 1, I explored the roles and methodological approaches of media arts 
practitioners as currently (and conflictingly) defined by a range of media artists 
and media arts engineers. I showed how much of the literature about the media 
arts presents approaches that are not always effective for media arts practitioners 
working in collaborative interdisciplinary research, or art-science, projects. I then 
presented a new methodological framework that included a range of tasks more 
effective for these kinds of collaborative research projects. 
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In Chapter 2, I examined a series of multimedia art installations that I helped 
to create, either as a collaborator or as the primary author, including Ccll Tango, 
Data Flow, and Information Poems. I made the argument that media arts projects, 
even those with solcly artistic outputs, are engaged, at least informally, with a 
range of research topics. Specifically, I looked at projects that explored research 
in Computer visión, information retrieval, visualization, interaction, natural lan- 
guage processing, among other topics. A main point of this chapter was to show 
that media arts projects, although seen as taking from existing research, could 
easily be seen as producing research. In particular, I discussed tenets of scientific 
methodologies and showed that media arts methodologies are not, as is commonly 
thought, in opposition to scientific research, and pointed to ways in which media 
arts projects could support these methodologies. 

In Chapter 3, I introduced the media arts projects, Fluid Autómata and An- 
nular Genealogy, that were intended to have both artistic and research outputs. I 
showed that, through a series of implementations and revisions, media arts project 
can function to generate new research questions, and similarly, that implementa¬ 
tions of research can be used to generate new artistic concepts. 

In Chapter 4,1 introduced a series of ongoing interdisciplinary research projects, 
including the Natural Material Browser and Augmented Communication Tools, 
that involved domain experts from múltiple disciplines. The methodological 
thernes discovered through my own work (in Chapters 2 and 3) and coalesced 
via my research in Chapter 1 was shown to be useful in a wide variety of situa- 
tions. 

In this final chapter, I summarize the specific contributions of the disserta- 
tion, make recommendations for media arts practitioners interested in working in 
art-science collaborations, and point to future directions for research in effective 
methodologies for media arts practitioners. Finally, I discuss the IEEE VIS Arts 
Program that I co-chaired in 2013 (and will chair again in 2014) and introduce 
current areas of active research in the field of visualization that are related to the 
concerns of this dissertation. 

This dissertation has presented the following main contributions: a comprehen¬ 
sivo literature review of media arts roles, methodological approaches and specific 
research methods; a methodological framework and specific tasks associated with 
the framework that enable media arts practitioners to more effectively engage 
in interdisciplinary projects; description of the methodological thernes, augmen- 
tation, generation, provocation, and mediation; examplcs of media arts projects 
whose artistic output is related to research in a variety of different areas; case 
studies where methodological approaches were instrumental in the development 
of effective interdisciplinary project; and the articulation of new areas of media 
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Main Contributions of the Dissertation 


Chapter 1 


1. Summarized methodological approaches in the literature of media arts. 

2. Summarized roles of media artists from the literature of media arts. 

3. Discussed how these thernes and roles are incomplete, personal, and 

contradictory. 

4. Synthesized new descriptions of media arts activities. 


Chapter 2 

1. Showed that arts projects share concerns of research projects. 

2. Showed that arts projects can be more than ornamental, aesthetic, 

or conceptual. 

3. Indicated that media artists could contextúa, li ze their work in terms 

of the research that they parallcl. 

4. Showed that media art-making can generate new areas of inqniry. 

5. Showed how media arts methodologies can directly support scientific 

methodologies. 


Chapter 3 

1. Presented a media arts project that had outputs in arts and research. 

2. Showed how media arts projects delinéate new areas of research. 

3. Explored how media arts projects offer new insight into collaboration. 


Chapter 4 


1. Presented examples of successfnl interdisciplinary research projects. 

2. Showed how representation and interaction can augment interdisciplinary 

research. 

3. Explored how mediation tasks can enable interdisciplinary research. 

4. Introdnced the idea of mediation tools as an area for fnrther exploration. 


Tablc 5.1: List of contributions per chapter. 
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arts research based upon these methodological themes. I explored and analyzed 
general themes in the literature of media arts about the role of media artists. I 
examined ways in which these themes are incomplete, personal, and contradictory. 
Thcn, from these themes, I synthesized what I find more accnrate descriptions of 
media arts activities, especially as they relate to larger interdisciplinary projects. 
I showed that even media arts projects that do not have explicit research goals 
share many concerns of projects that are explicitly contextualized as having re¬ 
search outcomes. And I showed that media projects that create large scale artifacts 
(installations, software, etc) are rigorously engaged in implcmentations that are 
much more than ornamental, aesthetic, or conceptual. I also indicated that media 
artists could and perhaps should contextualize their work in terrns of the research 
that they parallel. And I showed how media art-making is not simply “taking” 
from existing things (the bricolage model), but is also, potentially, generating 
new areas of inquiry. I also presented examples of media arts project that have 
múltiple outputs in both the arts and in research. I explored how media arts 
methodologies can, potentially, directly support scientific methodologies. And I 
explored specifically how media arts projects engaged with new forrns of visual- 
ization and interaction can offer new insight into collaboration. I also looked at 
examples of interdisciplinary research projects involving more than one person 
and showed how the media arts focus on representation and interaction can aug- 
rnent research projects in other domains. And I also discussed how media artists 
have an intrinsic awareness of how to bridge múltiple fields, and thus can be effec- 
tive mediators between múltiple domain experts, even in areas not traditionally 
associated with media artists, that is, not representational or interactive. Tablc 

5.1 lists the contributions that were rnade in each chapter. 

5.2 Visualization Model 

The fielcl of information visualization is concerned with issues regarding visual 
representations that enable effective reasoning about abstract data [154, 225]. It 
is generally situated within the larger discipline of Computer Science, focusing 
on advancing scientific knowledge through empirical methodologies and the use 
of precise evaluation metrics. Information visualization is inherently interdisci¬ 
plinary, and involves research in human-computer interaction, cognitive Science, 
computational methods, and graphic design. Although many of the media arts 
projects I discuss in this dissertation are related to topics in a range of research 
areas, here I use information visualization as an examplc to demónstrate how me¬ 
dia arts practitioners can serve as a bridge between the arts and research and to 
discuss the potential for enabling effective interdisciplinary collaboration. 
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The methodological themes I introduce in this dissertation in some ways par- 
allcl a modcl that has proved useful to the information visualization community. 
Tamara Munzner’s widely cited “Nested Model for Visualization Design and Val- 
idation” aróse out of a recognition that existing methodologies were not suffi- 
cicntly accurate in characterizing visualization research, and thus less effective at 
“guiding” researchers in defining the scope of their investigations and in making 
decisions [155]. 

According to the tenets of the Nested Model , an effective visualization research 
project should: a) accurately describe the specific domain targeted and correctly 
characterize the problems and data associated with that domain; b) map the prob- 
lerns and data into appropriate tasks and data types (what she calis operations ); 
c) design visual encodings and interaction techniques to support those operations; 
and d) create algorithms to make those encodings and techniques more efficient. 

For Munzner, an issue is that visualization research doesn’t always address 
each of these “levels” properly, or makes incorrect assumptions about how their 
research relates to them. Specifically, she identifies threats associated with each 
level, and also presents guidelines for handling these threats, and for providing 
validations for research activities. At the domain level, a potential threat is that 
the researcher mistakenly identifies a minor problem that is not necessary to solve, 
and ignores the major issues that actually are important to that community. A 
possible validation is to observe and interview domain experts who can give you 
insight into the problems in the domain. At the task level, one threat is poorly 
translating the problems into operations that might not effectively represent or 
address the domain problems. Here, a potential validation is to test the technique 
on target users and to conduct field studies to confirm that the tasks actually 
are useful for the particular problems. At the encoding and interaction level, 
a threat is that, even if the appropriate operations are identified correctly, the 
specific choices for visual represent at ion or choice of interaction techniques may 
be ineffective. In order to validate these choices, the researcher needs to justify 
their design decisions and also to conduct quantitative and qualitative user studies. 
Finally, at the algorithms level, a threat might be a slow or otherwise ineffective 
implcmentation of even a successful interaction technique or visual encoding. A 
validation could be to investígate the computational complcxity of the algorithm, 
or to quantitatively measure the rnethod to evalúate its effectiveness. 

Munzner recognizes that any visualization problem spans múltiple levels. In 
her model, each of these levels are interconnected by “upstream” dependencies 
and “downstream” influences. For instance, it may not be possible to validate the 
appropriateness of a task until a visual encoding is chosen and evaluated. And 
then that visual encoding might not be able to be evaluated until an algorithm is 
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designed that is fast enough to generate the visual encoding at interactive rates. 
Because each of these levels are interconnected, Munzner belicves that it is import 
to itérate over prototypes that test assumptions at each level before committing 
to a decisión about any one of the levels. Through analyzing the methodologi- 
cal requirements of a successful visualization project, Munzner provides effective 
guiclclines for how to think about a problem and how to approach the creation of 
new projects. It may in fact be lcss important whether or not the levels accurately 
carve out the levels of abstraction related to visualization research with complete 
precisión. 

5.2.1 Interdisciplinary Threats and Validations 

Although the “space” of interdisciplinary research is in many ways at once broader 
and less defined, and because media arts practitioners have a different set of priori- 
ties, it is not as straightforward to define specific levels for media arts practitioners 
that specifically enable pragmatic outcomes, maintain Creative agendas, and allow 
all participants to be directly integrated with the primary goals of interdisciplinary 
research collaborations. However, 1 do bclieve that it can be uscful to think in 
terms of threats to and validations of methodological thernes and also to iden- 
tify upstream dependencies and downstream implications of each of the thernes. 
A difference however is that, unlike in the nested model, which is explicitly, if 
loosely, hicrarchical, the guiclclines offered by these methodological thernes are 
more modular, and perhaps could be described as rhizomatic [46]. Nonetheless, I 
can present somc initial, incomplete threats and validations for each of the thernes, 
and indicate upstream and downstream connections to the other thernes, meant 
to be hclpful guiclclines without being overly prescriptive. 

5.2.1.1 Augmentation 

The goal of augmentation is to develop or utilize techniques that e n h an ce research 
goals. Its operations inelude: tasks relating to immersion, interaction, visualiza¬ 
tion, and sonification, or the creation of techniques or Systems that assist scientific 
methodologies. Threats to the successful integration into an interdisciplinary col- 
laboration inelude inadvertently working on tasks that are interesting rather than 
useful, and focusing on techniques that are original rather than effective (i.e., the 
“not invented here syndrome” [112]). Downstream implications arise from result- 
ing new understandings of data that may necessitate the need to generate new 
research. 
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5.2.1.2 Generation 

The goal of generation is to discover new Unes of research and to clarify tasks 
needed to reach research goals. Its operations inelude conceiving of and developing 
artworks that investígate interesting or not wcll defined topics in order to discover 
and promote novel research. A main threat is that this process can be seen as risky, 
that it ends up being counter-productive, or that it takes too long and expends 
too much energy. Another threat to interdisciplinary research projeets is that 
the team (or some of its members) becorne too focused on artistic outputs and 
neglect their relation to the research goals. One downstream implication is that 
the generation of new tasks and research ideas may necessitate critical cliscussion 
and further provocation about the research. 

5.2.1.3 Provocation 

The goal of provocation is to expose blind spots and to challenge assumptions 
in order to create new possibilities. Its operations inelude asking probing ques- 
tions and thinking about “broader impaets” and cultural implications seriously. 
Threats to interdisciplinary collaboration might inelude asking questions that are 
antagonistic rather than hclpful, that expand the scope of the project too far, 
that are too broad to be addressed effectively or within a suitable timeframe, or 
that are simply not applicablc to a particular research agenda. A downstream 
implication of provocation is that it introduces new ideas and people that require 
new forrns of mediation. 

5.2.1.4 Mediation 

Goals of mediation inelude effective knowledge sharing and the prioritizing of re¬ 
search goals. Its operations inelude developing mediation support tools, finding 
strategies to enable knowledge sharing, and exposing underlying rnodels of users, 
systems, and disciplines. Threats inelude the unnecessarily complication of com- 
munication and being unaware of one’s own biases and promoting specific rnodels 
or agendas. Downstream implications inelude the need to generate new research 
in order to mediate effectively and to augment data and rnodels relevant to the 
knowledge of different users. 

5.3 IEEE VISAP: Art+Experiment 

In this section I describe ongoing work to find connections between the method- 
ological thernes described in this dissertation and the visualization research com- 
munities, represented by the different conferences associated with IEEE VIS (Vi- 
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sual Analytics Science and Technology (VAST), Information Visualization (Info- 
Vis), and Scientific Visualization (SciVis)). I was, along with the designer Lauren 
Thorson, a chair of the IEEE VIS 2013 Arts Program. I liad the opportunity to 
shape the program to focus not only on artworks that engaged topics in visual¬ 
ization, but also to introduce a forum for interdisciplinary researchers to present 
work or to introduce new thinking about the role of art and research. In the cali 
for entries for the Arts Program, titled Art+Experiment, I asked artists and re¬ 
searchers to think about the potential of artworks to act, figuratively or literally, 
as research experiments: 


What could it mean for an art installation to produce experimental 
results? Can an artwork be expressive, challenging, and conceptual, 
yet simultaneously rigorous, practical, and empirical? We invite artists 
and researchers to think about the connections and chasms between 
art and research, and to explore the nature of experimental design and 
Creative experimentation [67]. 

Discussions about the relation of art and visualization are often centered on the 
usefulness of aesthetics for functional purposes [122], The influence of Edward 
Tufte’s seminal work contributes to the awareness that design and aesthetics 
are primary components of effective visual communication [210]. On the other 
hand, some writers and artists resist a too-literal reading of that understanding. 
For instance, Warren Sack explores the aesthetic appropriation of visualization 
metaphors and methods for artistic, rather than purely functional, purposes [182], 
He finds that many works of data visualization art are, in fact, relevant, chal¬ 
lenging, and even sublime. Seeking to delinéate the extent of what they cali 
“information aesthetics,” Andrea Lau and Andrew Van de Moere have designed 
a helpful domain rnodel that positions visualization projects on along continu- 
ums of focus: data, interaction, and aesthetics [124], In their view, work that 
emphasizes aesthetics has an “interpretive,” rather than a “direct” purpose, yet, 
however, successful visualizations could incorpórate both of these outlooks simul¬ 
taneously. Intriguingly, they stress that it is more useful to think about the term 
aesthetics as involving reasoning about context instead of limited to the realrn of 
representation. 

Just as interaction, analytics, and evaluation are integral concerns of visu- 
alization research, similarly visual representation is only one of many possible 
components of works of new media art. As the art historian Edward Shanken 
discusses in his investigations into the beginnings of new media arts and concep¬ 
tual art in the 1960s and 70s, media artists have always focused on Systems and 
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concepts more than on representations and technology, even as they make use of 
advances in representational technologies [193]. Although the phrase “experimen¬ 
tal art” tends, popularly, to refer to art that is avant-garde or perhaps simply 
inscrutable, much of the art by early media artists consciously characterized itself 
as intentional, quasi-scientific experiments in creating “meta-critical” experiences, 
demanding “that the viewer examine the process of processing information, while 
in the process of doing so” [191]. 

Much recent visualization research is concerned with the creation, implcmen- 
tation, replication, and evaluation of user studies in order to increase knowledge 
in the fields of information and scientific visualization. While these studies are 
often rneant to validate or reinforce the primary claims of novel visualization 
Systems and methods, there is a growing awareness that creating empirical ex¬ 
periments that lead to new knowledge (or a validation of new knowledge) is not 
necessarily a straightforward task, and that the effective design of visualization 
experiments itself requires research. For instance, as described above, Tamara 
Munzner’s “nested model” helps researchers untangle the many approaches to- 
ward validating visualization tasks [155]. Similarly, an artiele by visualization 
researcher Michael Sedlmair, et ah, outlines the complexity of choosing appro- 
priate design studies to convincingly evalúate visualization research [189]. And 
Ben Shneiderman calis into question the effectiveness of the “oíd strategies of con- 
trolled studies,” preferring instead to make use of more flexible, iterative methods 
that “embrace ethnographic styles of observation” [195]. 

Despite perhaps some complexity and confusión regarding the meaning of the 
word “experiment” in different disciplines, it is clear that experiments function 
differently in artistic and research contexts. Nonetheless, the hope is to provicle a 
Creative forum to foster dialog about possible connections between artistic exper¬ 
iments and research experiments. Moreover, I believe that thinking more deeply 
about these connections could lead to novel Creative works engaged in data and vi¬ 
sualization, as wcll as to starting investigations into Creative ways to design studies 
that validate research. In other words, I wanted to promote innovation in aesthet- 
ics and representaron but that simultaneously have an analytic component; that 
ask meta-level questions about the way in which aesthetics and representation 
function. Artists and researchers were encouraged to submit work that offered 
novel explorations of what various modalities of art might have to say about the 
design and meaning of experiments. While the Arts Program submissions repre- 
sent a wide range of perspectives and topics - and while they are certainly engaged 
in representational aesthetics and design considerations - the accepted works in 
the art show and art papers track are also concerned with the idea of art or artistic 
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perspectives as experiments that contribute to thinking creativcly about research, 
that are enablers of research, or that function as a form of research in and of itself. 

5.3.1 The Arts Program Submissions 

A number of thernes were evident across the Art Show submissions, including a 
focus on real-time streams of data, the use of immersive, participatory environ- 
rnents, and an interest in presenting provocative, qualitative modalities of data 
representation [66]. Three pieces engaged with textual data arriving in real-time 
via Twitter feeds: Byungkyu Kang presented an ambient portal of currently trend- 
ing topics using a “Tweet Probe” [111]; Ye Lin and Romain Vuillemont presented 
a novel Creative ”spirograph” representation of the Twitter feed for the CHI2013 
conference [133]; and a large, dynamic sculpture created by Hilary Harp and Barry 
Moon ernits sounds based on a sentiment analysis of incoming tweets. Another 
piece that engages with a different type of temporal data was presented by the 
Los Angeles based artist, Xárene Eskander. Her piece, “Saltón Sea Revisited,” 
presents her “realtime lapse” technique, concurrently presenting múltiple slices of 
video taken at different times across a single landscape [61]. 

The exploration of data within immersive spaces was also a popular theme this 
year. Yuan-Yi Fan, F. Myles Sciotto, and JoAnn Kuchera-Morin demonstrated a 
versión of their immersive installation, “Time Giver,” that measures and interprets 
EEG and PPG signáis, visualizing and sonifying thcm within 3D space [62], Qian 
Lili and Yoon Chung Han presented an interactive visualization using 3D are 
diagrams to explore the borrowing patterns of library visitors checking books 
in and out over the course of a year. And Jeong Han Kim presented “Qualia 
Landscapes,” visualizations made by analyzing data associated with different cities 
using his “emotional search engines” to characterize the mood of a city. Chin-En 
Keith Soo was also interested in Creative ways of transforming data. His work, 
“VICISS” is an interactive mirror than turns a user’s image and movements into 
melodic patterns. Many of the artists explicitly presented their work as an ongoing 
process of conducting artistic experiments. For instance, Yeohyun Ahn applied 
generative art to typography, using procedural texturing algorithms to evolve 
organic material into letters and words imbued with emotion and meaning [1], 
And Philip Galanter, displaying excerpts from the latest incarnation of his project, 
“XEPA,” presented a series of intelligent sculptures that evalúate each other’s 
output, each making high lcvcl aesthetic decisions in dialog with its neighbors. 

In addition to these artworks that incorpórate various modalities of interacting 
with, representing, and transforming data, the Arts Program also provides a forum 
for artists and researchers to engage with the unifying theme, Art+Experimenta 
through presenting in the papers track. Three papers explicitly discuss the rcla- 
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tionship of art to Science, or what is often termed “art-science” or “Art-Sci”. In 
the paper, “Art and Science as Creative Catalysts,” Elcanor Gates-Stuart and her 
colleagues explore a group of projects involving scientists, engineers, and artists, 
and offer strategies for how to intégrate the different modes of thinking related 
to their disciplines [86]. Similarly, Francesca Samsel provides an overview of 
what she defines as representative works that bridge art and Science [183]. And 
Ruth West, in collaboration with Roger Malina and others, explores the “crisis in 
representation” stennning from the ever-increasing size and complcxity of data; 
introducing the idea of “data remixing” as a potentially transformative approach 
to interdisciplinary collaboration that can inelude múltiple inputs and objectives 
[227]. Francis Márchese takes an historical approach, analyzing early experiments 
by medieval scholars who operated in some way as proto-data visualizaron re- 
searchers, creating the first known examples of certain types of abstract data 
representations [145]. More contemporarily, Julián Heinrich and Daniel Weiskopf 
present a novel rendering technique for obtaining visually pleasing images from 
parallcl-coordinate plots via “density footprints” [99]. Finally, Philip Galanter dis- 
cusses the role of evaluation in computational aesthetics, and, using his art show 
submission as an example, explores evolving Systems to mimic Creative processes 
[82]. 

The goal of the conference is for a continued presence and greater integration 
of the Arts Program with other conferences, symposia, and workshops that make 
up IEEE VIS, and especially that the introduction of the VISAP papers track will 
be conducive to mcaningful dialog about the role of art and creativity in scientific 
research, as well as the Creative potentials of introducing empirical methodologies 
into artistic practices. 

5.4 Future Directions 

This dissertation presents many ways in which media arts projects could be ben- 
eficial to interdisciplinary research project, and looks at projects that have both 
artistic and research outputs. However, in the penultimate chapter, I describe 
research projects that have solely research outputs. In my most recent collabora- 
tive work, although I am utilizing the methodological thernes I introduce, there 
have not been associated artistic outputs. That is, in one sense, I have the inverse 
of the problem I began with. A major future clirection that follows from this 
dissertation is to continué to explore how to be simultaneously effective in both 
domains. In particular, I will pursue and evalúate media arts projects that can 
serve as generators that help finel and focus interdisciplinary research topics. 


119 



Chapter 5. Conclusions 


Other future clirections that follow directly from the work in this dissertation 
inelude the idea of mediation as a primary focus for media artists to contribute 
to and to think about. For instance, in the Augmented Communication Tools 
project I described the idea of using interactive visualization as a way to explain 
the model of how a computational systern carne to its results, rather than those 
results in and of themselves. 1 believe that thinking of media arts projeets as 
explicitly engaged with transmitting rnodels for reasoning about the world is a rich 
topic requiring further exploration. Moreover, similarly to the way in which Ben 
Shneiderman positions the idea of creativity support tools as a way to reframe the 
goals of information visualization, the idea of “mediation support tools” rnight be 
a uscful way to think about creating projeets that enable effective interdisciplinary 
research between dornain experts in fields with widely divergent perspectives. 

In sum, this dissertation clarifies sorne of the practical roles, agendas, and tasks 
that are important and effective for media artists working within interdisciplinary 
collaborative projeets. I hope I have demonstrated how media arts practitioners 
can and do contribute to scientific endeavors and that I have provided a clear 
set of methodological themes with which media artists can draw from in order to 
guicle and articúlate their contributions in these areas. 
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